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Abstract 

Air  Force  policy  states  the  fundamental  reason  for  participating  in  technology  transfer 
is  to  maximize  the  return  on  investment  (ROI)  on  research  and  development  (R&D)  funds. 
Public  law  dictates  that  federal  agencies,  including  the  Air  Force,  are  to  spend  no  less  than 
0.5%  of  their  overall  R&D  budget  in  the  pursuit  of  technology  transfer.  However,  there  is 
currently  no  ROI  model  available  to  the  decision-maker  in  the  evaluation  of  alternative 
transfer  opportunities.  This  research  effort  develops  a  model  that  measures  the  ROI  of 
individual  cooperative  research  and  development  agreements  (CRD  As)  on  the  basis  of  the 
objective  and  subjective  benefits  amassed.  The  model  results  assist  the  decision-maker  by 
providing  a  relative  ranking  of  each  transfer  opportunity  in  comparison  to  one  another.  A 
sensitivity  analysis  method  and  results  are  included  which  identify  definite  regions  of 
alternate  “optimal”  choices  depending  on  the  weight  given  to  objective  and  subjective 
benefits.  Consequently,  the  decision-maker  is  provided  with  a  flexible  model  for  use  in 
maximizing  ROI,  the  Air  Force’s  goal  for  technology  transfer. 
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A  RETURN  ON  INVESTMENT  MODEL 
FOR  AIR  FORCE  TECHNOLOGY  TRANSFER 
I.  Introduction 

Background 

For  a  country  to  prosper  over  the  long  run,  its  economy  must  grow.  According  to 
President  Clinton  and  Vice  President  Gore,  “technology  is  the  engine  of  economic 
growth”  (Clinton  and  Gore,  1993:7).  Why  is  technology  so  important?  Technology 
improves  individual  Quality  of  life  and  promotes  a  nation  s  economic  strength  (Clinton  and 
Gore,  1993:2).  Without  this  engine,  the  country’s  economy  and  individual  quality  of  life 
cannot  flourish.  The  United  States  is  a  case  in  point,  as  two-thirds  of  its  productivity 
growth  since  the  Great  Depression  is  directly  attributable  to  technological  advancement 
(Clinton  and  Gore,  1993:7). 

Unfortunately,  the  US.  has  lost  some  of  the  large  technological  edge  it  once  enjoyed 
(Winebrake,  1992:54).  This  can  be  attributed  to  several  factors.  First,  the  fall  of  the 
Berlin  Wall  and  the  end  of  the  Cold  War  have  required  the  U.S.  to  redirect  a  technology 
strategy  which  had  been  unchanged  in  over  forty  years.  Second,  the  governments  of  other 
countries  are  actively  pursuing  technological  strategies  synergistically  linking  government 
to  private  industry.  Finally,  in  conjunction  with  the  previous  factor,  the  transfer  of 
technology  from  United  States  government  to  its  private  sector  is  lacking. 
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Fall  of  the  Berlin  Wall/End  of  Cold  War 

The  fall  of  the  Berlin  Wall  and  the  end  of  the  Cold  War  have  had  a  large  impact  on  the 
United  States’  approach  to  technology  development.  During  the  Cold  War,  the  United 
States  government’s  role  in  the  development  of  technology  was  limited  to  the  basic 
sciences  and  mission-oriented  R&D  (Clinton  and  Gore,  1993:1).  The  Department  of 
Defense  (DoD),  the  National  Aeronautical  and  Space  Administration  (NASA),  and  other 
agency  mission-oriented  research  focused  on  combating  Eastern  Europe  and  the  former 
Soviet  Union  (Clinton  and  Gore,  1993:1).  However,  change  is  required  if  the  United 
States  is  to  regain  its  former  position  of  strength  as  changing  times  require  changing 
policy.  In  accordance,  the  President  and  Vice  President  outline  the  new  role  of 
government  in  the  development  of  technology  in  their  document  entitled,  Technology  for 
America  ’ s  Economic  Growth,  A  New  Direction  to  Build  Economic  Strength.  The  primary 
tenets  of  the  new  direction  are: 

•  Strengthening  America’s  industrial  competitiveness  and  creating  jobs; 

•  Creating  a  business  environment  where  technical  innovation  can  flourish  and 
where  investment  is  attracted  to  new  ideas; 

•  Ensuring  the  coordinated  management  of  technology  all  across  the 
government; 

•  Forging  a  closer  working  partnership  among  industry,  federal  and  state 
governments,  workers,  and  universities; 
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•  Redirecting  the  focus  of  our  national  efforts  toward  technologies  crucial  to 
today’s  businesses  and  a  growing  economy,  such  as  information  and 
communication,  flexible  manufacturing,  and  environmental  technologies;  and, 

•  Reaffirming  a  commitment  to  basic  science,  the  foundation  on  which  all 
technical  progress  is  ultimately  built. 

The  overall  thrust  is  summed  up  by  saying  the  United  States  must:  (1)  refocus 
technology  development  to  those  items  needed  today  (in  the  post  Berlin  Wall/  Cold  War 
era);  (2)  establish  relationships  between  government  agencies,  as  well  as  between 
government  and  industry;  and  (3)  remain  committed  to  basic  science,  the  foundation  of 
technological  development  in  the  future. 

International  Technology  Strategies 

Much  of  the  United  States’  decline  in  technological  superiority  stands  in  stark  contrast 
to  the  approach  to  technological  development  taken  by  other  countries.  As  highlighted 
above,  the  United  States  is  in  the  midst  of  changing  its  policy  to  one  which  other  countries 
have  already  implemented.  The  governments  of  European  countries  have  been  enabling 
many  companies  to  prosper  via  government-funded  research  and  development  (R&D)  for 
years  (Scott,  1993:64).  Data  from  1992  indicate  that  the  number  of  Technology 
Application/Education  Centers  in  West  Germany  was  over  fifty,  Japan  had  over  one 
hundred  and  seventy,  while  the  United  States  had  less  than  ten  (Strategic  Development 
Staff,  1992: 1,  8).  This  is  especially  impressive  in  West  Germany,  a  country  going  through 
so  many  changes  after  the  fall  of  the  Berlin  Wall. 
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France,  Germany,  and  Italy  are  specifically  highlighted  as  countries  in  which  federal 
laboratories  and  private  industry  routinely  share  information  (Scott,  1993:64).  This 
sharing  results  in  the  development  of  commercial  products  (Scott,  1993:64).  These 
countries  and  their  private  industries  recognize  that  technology  transfer  has  the  potential 
to  be  a  “win-win”  situation. 

United  States  Technology  Strategy 

As  illustrated  by  other  countries’  success  and  the  new  policy  presented  by  the  United 

States’  President  and  Vice  President,  the  transfer  of  technology  from  government  to 

industry  is  vital  to  a  country’s  economic  success.  The  United  States  has  approached  this 

process  passively  in  the  past,  hoping  that  technology  developed  for  space  and  national 

defense  would  “leak”  over  to  the  private  sector  (Clinton  and  Gore,  1993  :7).  However, 

today  there  is  an  active  means  in  place  to  accomplish  this  process  called  technology 

transfer.  Technology  transfer  is  defined  by  the  Air  Force  Materiel  Command  (AFMC) 

Technology  Transfer  handbook  (1995:0-5)  as: 

A  process  by  which  facilities,  equipment,  or  other  resources  relating  to 
scientific  or  technological  developments  of  a  federal  laboratory  are  provided 
or  disclosed  by  any  means  to  another  industrial  organization,  including  a 
corporation,  partnership,  limited  partnership,  or  industrial  development 
organization;  public  or  private  foundation;  nonprofit  organization,  including 
a  university,  or  other  person  to  enhance  or  promote  technological  or 
industrial  innovation  for  a  commercial  or  public  purpose. 

The  current  tenor  of  technology  transfer  is  to  be  more  proactive,  to  actively  seek  and 

establish  relationships  between  government  and  private  industry,  in  effect  to  emulate  those 

countries  who  have  shrunk  the  technology  gap  that  once  existed  between  them  and  the 

United  States. 
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Technology  transfer’s  importance  increases  with  each  passing  year.  Why?  Because 
private  industry  is  “underinvesting  in  R&D”  due  to  a  perception  that  it  is  too  risky 
(Winebrake,  1992:54).  In  fact,  some  leading  companies  have  determined  that  R&D 
should  be  considered  a  sunk  cost,  not  an  investment  with  the  potential  to  bring  positive 
returns  (Willett,  1992:96).  However,  a  “fair  sharing”  of  the  risk  (partly  via  cost-sharing) 
between  government  and  private  industry  (as  illustrated  by  European  countries)  promotes 
economic  growth  and  makes  strides  in  eliminating  this  perception  concerning  R&D  money 
(Clinton  and  Gore,  1993:5).  Cost-sharing  brings  the  commitment  of  both  parties  to  the 
transfer  because  investment  returns  are  only  experienced  when  the  project  works 
(Winebrake,  1992:59). 

Technology  transfer  is  important,  too,  because  those  receiving  federal  R&D  dollars 
must  realize  where  the  money  originated.  Just  as  the  corporate  investor  has  a  right  to  a 
return  on  his/her  investment,  so  does  the  American  taxpayer.  Baron  found  that  the  United 
States  government  does  not  perform  cost-benefit  analysis  to  determine  whether  research 
entered  into  will  be  profitable  (Baron,  1990:38).  Therefore,  taxpayer  return  on  investment 
is  not  addressed,  and  the  federal  government  is  not  fulfilling  its  responsibility  in  this 
regard.  Taxpayers  provide  support  to  over  700  laboratories  and  pay  the  salaries  of  nearly 
one-sixth  of  the  nation’s  scientists  and  engineers  (100,000  total).  However,  less  than  5% 
of  the  30,000  patents  granted  to  the  federal  government  were  eventually  turned  into 
commercial  items  (AFMC  Handbook,  1995:A-1).  Mission-related  projects  are  being 
accomplished,  but  laboratories  have  the  ability  to  create  additional  benefits  via  technology 
transfer.  This  capability  must  be  exploited  for  two  reasons:  (1)  if  not,  the  technological 
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position  of  the  United  States  will  continue  to  erode;  and  (2)  the  perception  is  growing  that 
the  country  is  not  receiving  adequate  returns  on  its  R&D  investment  (Carr,  1992A:8). 
Relevancy 

There  is  much  regulation  and  guidance  addressing  technology  transfer.  So  why  has 
technology  transfer  been  so  limited?  The  lack  of  a  measurement  tool  is  one  critical  factor. 
There  exists  no  device  to  help  determine  which  technology  transfer  opportunities  should 
be  pursued.  There  is  no  dispute  that  technology  transfer  contains  benefits  and  costs. 
However,  politicians  see  benefits,  industry  sees  benefits,  but  R&D  managers  (those  left  to 
implement  technology  transfer)  only  see  costs.  The  reason  for  the  R&D  manager’s  view  is 
that  he  or  she  is  given  a  specific  amount  of  money  to  perform  his  or  her  mission  for  the 
year.  In  order  to  perform  technology  transfer,  the  resources  needed  (including  money  and 
manpower)  must  be  taken  from  that  “mission”  budget.  Without  the  ability  to  determine 
the  profitability  of  a  specific  technology  transfer,  R&D  managers  perform  what  they  get 
paid  for  (the  mission)  and  participate  in  technology  transfer  at  the  minimum  funding  level 
dictated  by  law  (0.5%  of  the  overall  R&D  budget)  (AFMC  Handbook,  1995:A-2). 

While  technology  transfer  costs  apply  to  federal  laboratories,  there  are  potential 
benefits,  too.  R&D  managers  realize  there  are  potential  benefits  such  as  royalties,  rental 
fees,  etc.  that  can  add  to  his  or  her  “pot  of  money.”  Once  again,  however,  the  lack  of  a 
measurement  mechanism  does  not  allow  these  potential  benefits  to  be  weighed  versus 
potential  costs.  This  is  especially  disheartening  when  one  considers  the  AFMC 
Technology  Transfer  handbook  states,  “The  Air  Force  participates  in  DT2  (domestic 
technology  transfer)  to  maximize  its  return  on  investment  in  RDT&E”  (AFMC  Handbook, 


1995:A-5).  This  statement  suggests  return  on  investment  is  the  purpose  for  conducting 
technology  transfer,  but  the  Air  Force  and  R&D  managers  currently  have  no  measurement 
tool  in  place  to  determine  return  on  investment.  Clearly,  this  issue  must  be  addressed. 

The  R&D  manager  of  each  federal  laboratory  is  in  a  compromising  situation.  On  the 
one  hand,  directorates  and  regulations  require  managers  to  foster  relationships  with 
private  industry  through  technology  transfer.  On  the  other  hand,  additional  funds  are  not 
provided  to  pay  for  the  resources  required  by  technology  transfer.  In  a  way,  just  as  the 
individual  investor  must  make  educated  guesses  as  to  where  to  invest  money,  the  R&D 
manager  must  do  the  same.  Presently,  though,  there  is  not  a  tool  in  existence  that  gives 
the  R&D  manager  the  ability  to  decide  which  technology  transfer  situations  to  enter. 

The  purpose  of  this  research  is  to  develop  a  return  on  investment  (ROI)  model  for 
potential  usage  by  those  holding  the  R&D  “purse  strings”  at  the  laboratory  level  in 
determining  which  technology  transfers  to  pursue.  This  model  will  measure  return  on 
investment,  a  step  that  is  necessary  to  achieve  the  Air  Force’s  purpose  for  being  in 
technology  transfer  in  the  first  place:  to  maximize  RDT&E  return  on  investment.  One 
cannot  maximize  return  on  investment  if  it  cannot  be  measured.  This  research  addresses 
this  problem. 

Research  Objectives 

The  research  objectives  of  this  thesis  are  related  to  the  simple  question,  “Which 
technology  transfer  opportunities  should  be  entered  into?”  In  order  to  answer  this 
question,  this  research  effort  will  evaluate  past  technology  transfer  performance  and  also 
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present  a  model  for  predicting  TT  performance.  More  specifically,  the  research  objectives 
are: 

A.  Propose  a  ROI  model  to  be  used  by  Air  Force  decision  makers  in  performing 
Technology  Transfer; 

B.  Evaluate  the  Return  on  Investment  (ROI)  of  past  Technology  Transfers; 

C.  Perform  Sensitivity  Analysis  on  Technology  Transfers  when  some  inputs  are 
uncertain. 

Thesis  Overview 

Chapter  II  provides  an  in-depth  review  of  research  concerning  technology  transfer  and 
return  on  investment  models.  Chapter  III  contains  the  methodology  of  this  research 
effort.  A  study  will  be  conducted  utilizing  information  collected  from  past  Air  Force 
technology  transfers.  In  Chapter  IV,  the  data  collected  on  previous  Air  Force  technology 
transfers  will  be  analyzed  according  to  a  ROI  model.  The  research  will  conclude  in 
Chapter  V  with  a  discussion  of  results,  conclusions  drawn,  and  areas  of  potential  follow- 
on  research. 
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n.  Literature  Review 


Introduction 

The  federal  government  was  expected  to  spend  $62.2  billion  on  R&D  in  1997,  more 
than  half  the  total  ($120.5  billion)  expected  to  be  spent  by  all  of  privateindustry  (Studt 
and  Duga,  1997:2).  According  to  Duga,  both  the  government  and  private  industry  have 
evolved  from  a  “just  in  case”  to  a  “just  in  time”  approach  concerning  technology 
development  (Duga,  1998:4).  “Just  in  case”  R&D  spending  (1960’s  -  1990),  was  highly 
focused  on  attempting  to  gather  basic  knowledge;  “just  in  time”  R&D  spending  (1990  - 
present)  is  focused  on  gamering  technology  based  upon  availability,  regardless  the 
supplier  (Duga,  1998:4).  With  this  current  situation  and  the  capabilities  of  the  federal 
laboratory  system,  there  is  much  opportunity  for  government  and  industry  cooperation  in 
the  development  of  technology.  However,  84%  of  private  industry  is  expected  to  contract 
out  their  R&D,  with  only  4%  directed  toward  federal  laboratories.  This  can  be  partially 
attributed  to  the  fact  that  federal  laboratory  R&D  managers  are  hesitant  to  enter  into 
technology  transfer.  Though  current  directives  require  proactive  measures,  R&D 
managers  Oust  like  the  rest  of  the  Department  of  Defense)  realize  that  the  mission  comes 
first.  If  actions  not  directly  related  to  the  mission  cannot  be  measured  as  to  their  impact 
on  funding  and  manpower  usage,  they  will  not  be  pursued. 

President  Clinton  and  Vice  President  Gore’s  1993  directive  is  not  the  only  guidance 
associated  with  technology  transfer.  In  fact,  it  is  only  a  small  part  of  a  long  procession  of 
regulations,  directives,  and  instructions  which  are  presented  in  this  chapter.  The  further 
purpose  of  this  chapter  is  to  address  past  research  on  the  topics  of  technology  transfer  and 
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return  on  investment  (ROI).  Much  research  has  been  done  on  the  topics  of  technology 
transfer  and  ROI;  however,  the  two  have  not  been  linked  successfully.  An  appropriate 
connection  of  the  two  topics  will  help  overcome  the  shortcomings  currently  existing  in 
technology  transfer  evaluation  and  participation. 

Technology  Transfer  Guidance 

Technology  transfer  is  not  a  new  idea.  Its  roots  go  back  at  least  as  far  as  1956  when 
the  Army-Navy  Instrument  Program  (ANIP)  provided  the  fundamentals  needed  for  the 
development  of  military  and  civilian  products  such  as  solid-state  computers, 
microcircuitry,  and  the  field  of  human  factors  (Scott,  1992:64).  However,  the  stream  of 
laws  and  regulations  began  in  1980. 

Technology  transfer  regulations  were  developed  in  hopes  of  optimizing  the  use  of 
federal  laboratories,  their  100,000  scientists  and  engineers,  and  the  $18  billion  spent  on 
R&D  at  the  time  (AFMC  Handbook,  1995:A-1).  The  University  and  Small  Business 
Patent  Procedure  Act  (also  known  as  the  Bayh-Dole  Act  of  1980)  attempted  to  increase 
the  use  of  government  technology  by  giving  non-profit  organizations  and  small  businesses 
the  ability  to  retain  patents  for  technologies  developed  via  government  funds  (AFMC 
Handbook,  1995:A-1).  Provisions  of  the  Bayh-Dole  Act  of  1980  were  extended  to  all 
government  contracts  by  Executive  Order  in  1983. 

The  Bayh-Dole  Act  of  1980  was  followed  by  the  Stevenson-Wydler  Technology 
Innovation  Act  of  1980.  This  Act  requires  the  government  to  ensure  “. . .  full  use  of  the 
results  of  the  nation’s  federal  investment  in  research  and  development”  (AFMC 
Handbook,  1995:A-2).  An  additional  provision  of  the  Act  includes  the  establishment  of 
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Offices  of  Research  and  Technology  Applications  (ORTAs),  whose  purpose  is  to 
disseminate  information  and  help  transfer  technology  from  the  laboratory  to  the  public  and 
private  sectors  (AFMC  Handbook,  1995:0-3).  The  Act  requires  each  federal  agency  to 
spend  no  less  than  0.5%  of  its  R&D  budget  on  technology  transfer-related  activities 
(AFMC  Handbook,  1995:A-2).  The  Act  also  mandates,  where  appropriate,  the 
encouragement  of  the  exchange  of  scientific  and  technical  people  among  academia, 
industry,  and  federal  laboratories,  something  that  is  acknowledged  to  have  never  been  fully 
implemented  (AFMC  Handbook,  1995:A-2). 

The  subsequent  piece  of  legislation  was  the  Federal  Technology  Transfer  Act  of  1986 
(FTTA  of  1986).  This  Act  is  an  amendment  to  the  Stevenson-Wydler  Technology 
Innovation  Act  (AFMC  Handbook,  1995:A-2).  The  key  provisions  are  as  follows: 

A.  Agency  and  laboratory  directors  have  latitude  to  enter  into  Cooperative  Research 
&  Development  Agreements  (CRD As),  and  to  negotiate  patent  licenses. 

B.  It  permits  royalty  income  from  patent  licensing  and  assignment  of  at  least  15%  of 
the  royalties  to  the  inventor. 

C.  Technology  transfer  is  made  a  job  requirement  of  every  federal  laboratory  scientist 
and  engineer. 

D.  ORTA  involvement  in  laboratory  management  development  programs  is  increased. 

E.  It  provides  a  home  for  the  Federal  Laboratory  Consortium  (FLC),  a  group  of  400 
federal  laboratories  from  1 1  separate  government  agencies. 

As  one  can  see,  an  overarching  theme  of  this  act  is  to  provide  incentive  for  laboratory 

personnel  to  participate  in  technology  transfer. 

Several  other  pieces  of  legislation  have  facilitated  technology  transfer.  Executive 
Order  12591,  Facilitating  Access  to  Science  and  Technology,  became  effective  April  10, 
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1987  (AFMC  Handbook,  1995:A-4).  It  reiterates  certain  provisions  of  the  FTTA  Act  of 
1986,  such  as  mandating  heads  of  executive  departments  and  agencies  to  promote  CRD  As 
to  the  extent  possible  by  the  law  (AFMC  Handbook,  1995:A-4).  The  order  also 
encourages  granting  titles  to  the  result  of  federally  funded  R&D  to  contractors,  regardless 
of  size,  in  exchange  for  royalty-free  use  by  the  government.  The  National  Defense 
Authorization  Act  for  FY  93  expanded  the  ability  of  federal  agencies  to  participate  in 
technology  transfer  by,  among  other  things,  directing  the  Department  of  Energy  to  issue 
guidelines  to  facilitate  technology  transfer  to  small  businesses.  The  Domestic  Technology 
Transfer  Program  Regulation  of  1988  defines  the  responsibilities  of  an  ORTA,  the  Under 
Secretary  of  Defense  for  Acquisition,  and  the  heads  of  DOD  Components  (AFMC 
Handbook,  1995:A-4). 

One  regulation  is  Air  Force  specific.  The  Air  Force  Domestic  Technology  Transfer 
Policy  Directive  61-3  implements  the  previous  directives  and  defines  the  purpose  of  the 
Federal  Domestic  Technology  Transfer  (DT2)  program  (AFMC  Handbook,  1995  :A-5). 
Specifically,  “the  Air  Force  participates  in  DT2  to  maximize  its  return  on  investment  in 
RDT&E”  (AFMC  Handbook,  1995  :A-5). 

The  regulation  process  has  evolved  over  time,  as  indicated  above.  The  regulations  are 
directive  in  nature,  but  contain  loopholes,  in  that  one  must  determine  if  technology 
transfer  is  feasible  in  many  situations.  What  will  make  an  organization  spend  more  than 
the  minimum  0.5%  of  the  R&D  budget  on  technology  transfer  efforts,  as  mandated  by 

law? 
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Technology  Transfer  —  Past  Research  Efforts 


Technology  transfer  is  defined  as: 

A  process  by  which  facilities,  equipment,  or  other  resources  relating  to  scientific  or 
technological  developments  of  a  federal  laboratory  are  provided  or  disclosed  by 
any  means  to  another  industrial  organization,  including  a  corporation,  partnership, 
limited  partnership,  or  industrial  development  organization;  public  or  private 
foundation;  nonprofit  organization,  including  a  university,  or  other  person  to 
enhance  or  promote  technological  or  industrial  innovation  for  a  commercial  or 
public  purpose. 

(AFMC  Handbook,  1995:0-5) 

A  thorough  review  of  the  literature  associated  with  technology  transfer  indicates  three 
streams  of  research:  the  technology  transfer  process  (and  ways  to  improve  the  process), 
technology  transfer  participants,  and  technology  transfer  measurement.  This  is  in  line  with 
the  definition  from  the  AFMC  Handbook,  which  addresses  the  transfer  process,  the 
organization  involved,  and  the  desired  outcomes  by  which  to  measure  the  success  of  the 
transfer. 

Technology  Transfer  -  The  Process/Process  Improvement 

Van  Egeren  notes  that  there  is  much  difference  of  opinion  over  what  steps  compose 
the  actual  technology  transfer  process  (1997:26).  At  its  most  basic,  the  technology 
transfer  process  initiates  with  the  technology  being  developed  by  a  federal  agency,  the 
technology  is  then  placed  into  the  hands  of  the  end  user,  and  the  process  is  completed  as 
the  technology  is  utilized  by  the  end  user  (AFMC  Handbook,  1995  :D-1).  The  basics  of 
this  process  are  similarly  defined  by  Souder,  Nashar,  and  Padmanabhan  (1990:5-6).  The 
authors  indicate  that  the  technology  transfer  process  involves  four  stages: 
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Stage  1- Prosnectine:  research,  analytical,  and  decision-making  activities 
aimed  at  screening  alternative  concepts  or  technologies; 

Stage  2-  Developing:  physical  and  laboratory  R&D  activities  focused  on 
enhancing,  elaborating,  and  tailoring  the  technologies  chosen  in  Stage  1  to 
meet  users’  requirements; 

Stage  3-  Trial:  testing  the  developed  technologies  in  the  field; 

Stage  4-  Adoption:  final  development,  modification,  and  implementation  by 
the  user  (Souder  et  al.,  1990:5-6). 

The  above  definitions  indicate  technology  transfer  is  a  fluid,  dynamic  process  in  which 
steps  may  occur  simultaneously,  systematically,  or  be  skipped  entirely  in  certain  instances. 
The  lack  of  a  well-defined  process  opens  up  a  plethora  of  opportunities  to  improve  the 
process,  as  the  following  authors  highlight. 

Papadakis  explores  the  role  of  government  in  the  technology  transfer  process  (1995). 
She  notes  that  the  government  previously  enticed  its  federal  laboratories  with  “carrots”  to 
perform  technology  transfer,  whereas  present-day  legislation  is  more  like  a  “stick” 
(1995:55).  In  other  words,  federal  laboratories  were  previously  encouraged  to  perform 
technology,  but  are  presently  mandated  to  do  so.  Of  the  $23  billion  in  federal  laboratory 
funds  spent  on  R&D  in  1993,  91%  belonged  to  five  organizations:  the  United  States 
Department  of  Agriculture  (USD  A),  the  Department  of  Energy  (DOE),  the  Department  of 
Health  and  Human  Services  (HHS),  the  National  Aeronautical  and  Space  Administration 
(NASA),  and  the  Department  of  Defense  (DOD)  (Papadakis,  1995:57).  Papadakis  points 
out  that  the  orientation  of  most  (80%)  of  the  federal  laboratory  system  is  government 
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mission-oriented  R&D,  which  makes  it  difficult  for  technology  transfer  to  succeed 
(1995:55).  With  this  in  mind,  the  author  states  that  technology  transfer  will  be  most 
successful  when: 

A.  The  original  R&D  is  undertaken  with  the  end  user  in  mind  or  at  the  end  user’s 
initiation; 

B.  Key  uncertainties  of  commercialization  are  eliminated; 

C.  There  is  a  strong,  existing  system  for  commercialization;  and/or 

D.  New  forms  of  institutional  arrangements  are  created  to  handle  commercialization 
and  technology  transfer. 

This  indicates  technology  transfers  entered  into  must  be  thoroughly  examined  as  to  their 
commercialization  potential  at  the  outset  of  the  process. 

Carr’s  research  indicates  two  primary  reasons  showing  the  importance  of  improving 
the  technology  transfer  process.  First,  there  is  a  widespread  perception  in  industry  and 
government  that  the  nation  is  not  realizing  an  adequate  return  for  its  substantial  investment 
in  the  federal  laboratory  system.  Second,  the  federal  government  is  looking  for  new  ways 
to  enhance  U.S.  competitiveness  in  the  wake  of  a  serious  deterioration  in  the  U.S.  trade 
balance  (1992A:8).  As  a  comparison,  Carr  compares  the  Massachusetts  Institute  of 
Technology  (MIT)  versus  the  Department  of  Energy  (DOE),  a  component  of  the  federal 
laboratory  system.  Table  I  reveals  the  results  of  the  comparison: 


Table  1.  MIT  &  DOE  Comparison 


MIT 

DOE 

$’s  spent 

$700  million 

Over  $7  billion 

Royalties 

2  times  that  of  DOE 

Vi  that  of  MIT 

Licenses 

Same 

Same 
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The  head  of  MIT’s  Office  of  Technology  Licensing  responds  to  these  results  by 
estimating  an  order  of  magnitude  increase  in  technology  transfer  from  federal  laboratories 
to  industry  is  feasible  at  the  given  levels  of  R&D  money  (1992A:9).  Carr  states  the 
discrepancies  are  related  to  differences  in  the  technology  transfer  processes  (1992A:9). 

In  addition  to  describing  the  motivation  for  technology  transfer  improvement  and 
providing  indications  that  there  is  room  to  improve,  Carr  provides  three  models  to  pursue 
in  improving  technology  transfer  (1992A:  16).  The  three  models  are  called  the  legal, 

administrative,  and  marketing  models  (1992A:  16). 

Legal  model  programs,  also  known  as  patenting  programs,  are  generally  run  by  the 
organization’s  legal  staff  and  focus  exclusively  on  patenting  inventions  (1992A:  16).  The 
motivation  behind  this  model  is  that  organizations  cooperating  with  the  government  on  a 
specific  transfer  of  technology  are  given  patent  rights  to  the  technology  being  transferred. 

The  second  model  is  the  administrative  model,  in  which  programs  in  universities 
generally  are  created  as  part  of  an  administrative  or  support  organization  or  are  attached 
to  a  school  or  department  with  active  R&D  programs  (Carr,  1992 A.  16).  Federal 
laboratories  began  leaning  in  this  direction  during  the  1980’s  technology  transfer 
legislation,  in  which  more  effort  was  placed  on  evaluating  a  technology’s  commercial 
potential.  In  addition,  there  was  an  increase  in  licenses  of  technology  (Carr,  1992A:16). 

The  final  model,  the  marketing  model,  appears  to  have  the  most  merit  for  present  day 
use  (1992 A:  16).  In  this  model,  technologies  are  actively  marketed  according  to  their 
potential  for  licensing.  Carr  says  there  are  two  primary  avenues  federal  laboratories 
should  explore  by  using  the  marketing  model.  The  first  is  to  market  available  technologies 


16 


in  a  way  similar  to  private  laboratories  and  universities.  The  second  opportunity  is  to 
target  the  large  segment  of  private  industry  that  is  unaware  of  federal  laboratory 
capabilities  or  are  concerned  with  cooperating  in  a  project  with  a  federal  government 
agency  ( 1 992 A:  16). 

In  the  second  part  of  his  research,  Carr  identifies  technology  transfer  best  practices 
(1992B).  Carr  implies  that  a  major  problem  with  the  federal  laboratories’  participation  in 
technology  transfer  is  the  lack  of  marketing  (1992 A:  16).  However,  he  recognizes  there 
are  some  limitations  to  the  federal  government’s  transferring  of  technology,  such  as 
national  security  (certain  pieces  of  information  are  not  appropriate  outside  the  confines  of 
the  federal  government),  U.S.  preference  (technology  should  be  transferred  within  the 
U.S.  first),  fairness  of  opportunity  (all  should  have  an  equal  opportunity  to  participate  in 
technology  transfer),  and  conflict  of  interest  (especially  concerning  the  participants)  (Carr, 
1992A:18-19).  Notwithstanding,  it  is  important  for  federal  laboratories  to  conduct 
marketing,  understanding  the  preceding  limitations,  as  it  is  recognized  that  simply  placing 
technology  on  a  shelf  is  not  an  effective  way  of  initiating  the  technology  transfer  process 
(Carr,  1992B:27). 

After  establishing  the  need  for  marketing  in  the  technology  transfer  process,  Carr 
identifies  several  approaches  to  marketing  (Carr,  1992B:27-28).  The  first  option  is 
entitled  the  “First-Contact  Marketing  Strategy”  (Carr,  1992B:27).  This  approach  consists 
of  having  the  inventor  or  technology  transfer’s  office  identify  potential  transfer  partners 
and  having  the  first  firm  that  demonstrates  an  ability  to  participate  become  the  partner. 

The  primary  benefit  is  the  ability  to  minimize  the  use  of  scarce  resources  in  finding  the 
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industry  participant.  This  is  MIT’s  approach,  which  as  stated  earlier  provided  them  the 

same  number  of  licenses  as  the  DOE  at  one-tenth  the  price  tag.  The  second  marketing 

strategy  is  called  “Focused-Publicity”  (Carr,  1992B:28).  The  primary  tenets  of  this 

strategy  include  the  use  of  press  releases  and  computer  bulletin  boards  to  make  the 

availability  of  the  technology  for  transfer  known.  There  is  no  conclusion  made  concerning 

which  strategy  is  better,  but  instead  the  importance  of  marketing  is  the  point  made. 

Different  situations  require  different  marketing  approaches. 

Weijo  discusses  several  strategies  available  to  promote  technology  transfer  (1987). 

Weijo  acknowledges  recent  changes  in  legislation  have  helped  the  technology  transfer 

process,  but  he  indicates  marketing  is  important,  just  as  Carr  argues  above.  Weijo 

discusses  three  different  strategies  and  their  associated  goals  (1987:44-45): 

1  DEMAND  PULL  -  passive.  Wait  for  the  customer  to  bring  ideas  to  you. 

GOAL:  make  information  available  to  individuals  and  organizations  looking  for  answers 
to  problems  customers  have  identified. 

2.  ROLE-DIRECTED-  active.  Private  industry  engineers,  etc.  must  make  the  leap 
between  what  government  technology  is  being  provided  and  how  they  can  use  it. 

GOAL:  make  people  inside  the  organization  aware  and  knowledgeable  about  the  research 
being  conducted  in  government-sponsored  laboratories. 

3.  ORGANIZATION  DIRECTED-  most  active.  Government  personnel  determine 
who  can  use  the  technology  and  how.  (start  to  finish  involvement) 

GOAL:  “active  communication  with  all  important  roles  involved  in  developing  and 
implementing  technologies  in  a  firm.” 

Another  way  to  look  at  this  is  according  to  what  Weijo  defines  as  the  technology  transfer 
continuum,  pictured  in  Figure  1 : 
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Diffuses  - 

Audience  Passive 

Role-Directed  Firm-Directed  Audience 

Figure  1.  The  Technology  Transfer  Continuum 


The  federal  agency  participating  in  technology  transfer  has  the  option  to  focus  the 
audience  on  a  specific  technology  it  feels  can  be  used  in  the  commercial  sector  (the  firm- 
directed  approach),  or  on  the  other  extreme  let  industry  come  to  them  with  suggestions 
and  start  from  scratch  (the  passive  approach).  Of  course,  the  transfer  process  can  be 
executed  according  to  any  point  between  the  two  extremes.  Once  again,  different 
situations  require  different  approaches. 

Winebrake  (1993)  studies  the  mechanisms  in  place  for  federally  funded  technology 
transfer.  Winebrake’ s  research  examines  the  results  of  1 16  case  studies  in  the  Department 
of  Energy  (1992:55).  The  first  step  in  technology  transfer  is  the  identification  of  the 
technology  to  be  transferred.  In  doing  so,  one  can  determine  how  to  market  the 
technology:  passively  or  actively.  According  to  Winebrake,  there  are  five  stages  of 
technology  development: 

1 .  Basic  R&D  -  new  knowledge  is  discovered 

2.  Exploratory  R&D  -  attempt  to  discover  an  unknown  application  of  technology 

3 .  Applied  and  Tnformation-Based  R&D  -  research  performed  to  apply  basic 
knowledge  to  a  particular  problem 

4  Technology  Development  -  R&D  is  hardware  oriented  and  the  application  is 
already  designed 

5 .  Market  penetration  -  effort  placed  on  finding  market  uses  of  the  technology 
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The  maturity  of  the  technology  when  private  industry  gets  involved  in  the  transfer  dictates 

the  stage  at  which  marketing  will  occur.  If  the  government  wants  to  “push  their 
technology  to  the  field,  private  industry  will  not  enter  the  process  until  market  penetration. 
If  the  government  is  allowing  the  market  to  “pull”  (drive)  which  technology  it  attempts  to 
commercialize,  private  industry  will  enter  the  process  much  earlier. 

In  addition  to  discussing  technology  stages  of  development,  Winebrake  identifies 
several  practices  that  enhance  the  “success”  of  transfers.  Success  was  measured  on  a  5- 
point  Likert  scale  (1  meaning  not  very  effective  to  5  meaning  very  effective)  by  the 
government  agency  member  primarily  responsible  for  the  transfer  (Winebrake,  1992.56). 
Winebrake  performed  t-tests  on  a  variety  of  factors  assumed  to  contribute  to  successful 
transfer  of  technology.  The  two  factors  found  to  be  significant  at  a  95%  confidence  level 
were  having  advisory  groups  and  collaboration  with  cost  sharing.  Advisory  groups 
(having  members  of  the  government  and  private  industry  participate  together  from  the 
outset)  were  seen  to  be  important  because  they  overcome  two  significant  barriers  to  the 
technology  transfer  process:  differing  expectations  among  the  two  parties,  and  a  lack  of 
awareness  of  the  value  of  the  technology  being  transferred  (Winebrake  1992:57). 
Collaboration  with  cost  sharing  is  important  for  four  primary  reasons  (Winebrake, 
1992:57): 

A.  Cost  sharing  (and  the  potential  financial  losses  associated  with  it)  forces  recipients 
to  look  more  carefully  at  a  project  before  entering  into  a  collaborative  agreement, 
and  only  those  projects  showing  high  success  potential  are  chosen, 

B.  Cost  sharing  may  attract  recipients  with  money  already  invested  in  the  technology 
elsewhere  or  in  some  other  form,  and  these  recipients  bring  with  them  the 
experience  that  successful  transfers  need; 
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C.  Cost  sharing  assures  certain  intellectual-property  rights  or  licenses  remain 
protected,  and  hence,  the  potential  profit  incentive  of  these  projects  remains; 

D.  Cost  sharing  breeds  a  commitment  by  both  parties  to  make  a  project  work  because 
only  successful  projects  bring  returns  to  the  investor. 

The  first  and  last  reasons  are  probably  the  most  important  as  they  require  both  parties  to 

examine  the  potential  technology  transfer  before  the  process  begins,  and  if  they  agree  it  is 

worthwhile,  they  are  required  to  invest  money  based  upon  this  decision. 

Once  both  parties  agree  on  the  technology  to  be  transferred,  they  must  enter  into  an 
agreement.  Government  legislation  has  created  what  is  called  a  Cooperative  Research  & 
Development  Agreement  (CRD  A).  CRD  As  were  first  authorized  by  the  1986  Federal 
Technology  Transfer  Act  (Knauth,  1991:33).  Knauth  states,  “A  CRD  A  is  a  business 
contract,  not  a  procurement  contract”  (1991 :3 5).  The  government  cannot  place  money 
toward  the  CRD  A,  but  can  provide  other  resources  such  as  people,  buildings,  and 
services.  CRD  As,  along  with  the  other  legislation  concerning  technology  transfer,  is  an 
about  face  from  the  government’s  previous  stance  on  technology  sharing.  Incentives  now 
are  in  place  for  private  industry  to  cooperatively  participate  with  the  government  in  the 
transfer  of  technology.  Such  incentives  include  patents  and  licenses  (Knauth,  1991:33). 
Some  incentives  are  in  place  for  government  personnel,  too.  Federal  researchers  can 
personally  receive  a  portion  of  the  royalties  the  government  receives  from  the  private 
company. 

Knauth  indicates  the  concept  of  CRD  As  is  a  good  one,  but  that  it  still  requires  some 
“fine  tuning”  (1991:38).  Psychological  barriers  associated  with  existing  prejudices  on 
both  sides  remain  as  CRD  As  and  associated  legislation  have  completely  reoriented 
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government  and  business  on  this  issue.  A  final  barrier  to  overcome  is  bureaucracy,  a 
common  complaint  about  the  government  in  general.  It  is  stated  that  CRD  As  often  take 
up  to  10  months  from  conception  to  signing,  an  unacceptable  time  frame  considering  the 
types  of  technologies  being  transferred.  The  potential  problem  is  that  competitors  may  be 
able  to  place  similar  products  on  the  market  before  the  companies  going  through  the 
CRDA  process  are  able  to  finish  this  administrative  procedure. 

A  1994  GAO  Report  provides  more  recent  documentation  concerning  CRD  As  within 
the  Department  of  Energy  (DOE).  The  purpose  of  the  study  was  to  identify  the  benefits 
associated  with  CRD  As.  It  was  noted  by  the  authors  of  the  report  that  costs  were 
completely  ignored.  The  study  examined  10  CRD  As  and  concluded  the  following: 

A.  The  CRD  As  offered  opportunities  for  federal  laboratories  and  industry  to 
collaborate  on  research  while  meeting  their  missions. 

B.  Technology  from  federal  laboratories  was  transferred  to  the  private  sector, 
resulting  in  commercial  products. 

C.  R&D  programs  were  advanced. 

D.  The  sharing  of  resources  aided  federal  laboratories  and  private  companies  in 
accomplishing  CRDA  objectives. 

These  results  indicate  that  CRD  As  are  helping  achieve  the  objectives  of  technology 
transfer,  at  least  for  this  small  sample. 

It  is  apparent  from  the  above  research  that  the  technology  transfer  process,  due  to  its 
fluidity,  requires  much  coordination  and  flexibility.  What  works  in  one  situation  doesn  t 
necessarily  work  in  another.  An  important  aspect  of  technology  transfer  is  understanding 


the  participants,  their  roles,  and  their  perceptions.  It  is  this  venue  of  research  that  is 
examined  next. 

Technology  Transfer  -  The  Participants 

The  success  of  a  specific  technology  transfer,  as  one  might  expect,  is  highly  dependent 
upon  the  participants.  Technology  transfers  are  partnerships  and  understanding  who  is 
involved  and  what  is  important  to  them  is  of  utmost  importance.  It  is  a  lack  of 
understanding  that  will  either  preclude  entering  into  transfer  opportunities  or  degrade  the 
success  of  transfers  pursued.  The  following  research  efforts  address  these  issues. 

Baron’s  research  is  concerned  with  overcoming  barriers  to  the  technology  transfer 
process;  a  primary  one  being  the  differences  among  participants  (1990:38).  At  the  most 
elemental  level,  there  are  two  participants  in  the  transfer  of  any  technology:  the 
government  and  the  end  user.  Baron  cites  the  loss  of  U.S.  leadership  in  technology  as  the 
reason  the  government  is  proactively  pursuing  technology  transfer  (1990:38).  However, 
companies  are  hesitant  to  participate,  partly  because  the  government  has  been  a  poor 
partner  in  the  past.  Examples  of  poor  government  follow-through  include  nuclear  fuel 
processing,  fast  breeder  reactors,  and  solar  and  wind  energy.  The  government  enticed 
private  industry  to  participate  in  these  technological  arenas,  only  to  back  out  (1990:38). 

In  order  to  overcome  the  past,  an  understanding  of  the  other  participant  and  its  culture  is 
important.  Baron  breaks  them  down  in  the  following  way: 
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Government 


1 .  Performs  research  for  research’s  sake,  not  with  a  particular  product  in  mind. 

2.  Not  concerned  about  confidentiality;  thus,  present  results  at  conferences  and  open 
forums. 

3.  No  cost-benefit  analysis  is  done  to  determine  whether  the  research  will  be 
profitable. 

4.  Research  is  long-term  in  nature. 

Industiy 

1 .  Short-term  missions  for  reasonable  returns  on  investment 

2.  Confidentiality  is  crucial. 

3.  Do  not  view  national  laboratories  as  part  of  the  “real  world”  due  to  their  lack  of 
attention  to  budgets  and  schedules. 

Having  an  understanding  of  the  above  issues  enhances  the  possibility  technology  transfers 
of  mutual  benefit  will  occur. 

In  his  research,  Roessner  examines  what  industry  wants  from  federal  laboratories 
(1993).  In  these  changing  times,  industry  needs  what  the  federal  laboratories  can  provide 
and  vice  versa.  Companies  are  feeling  more  pressure  to  look  for  outside  expertise  and 
technology  (Roessner,  1993:39).  In  fact,  84%  of  industrial  R&D  is  expected  to  be 
contracted  out  in  1997  (Studt  and  Duga,  1997:2).  On  the  other  hand,  federal  laboratories 
are  being  tasked  to  work  with  industry  more  via  the  stream  of  technology  transfer 
legislation.  These  factors  appear  to  provide  the  foundation  for  a  win-win  situation. 
However,  private  industry  has  reservations  about  entering  into  technology  transfer 
projects  with  the  government.  Private  industry  managers  suggest  a  major  contributor  to 
this  perception  is  that  government  does  not  know  what  industry  wants  when  it  comes  to 
technology  transfer.  Industry  admits  it  needs  help,  but  not  the  kind  of  help  the 
government  currently  provides.  Roessner  suggests  that  companies  want  leverage  on  their 
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own  R&D  projects,  not  market  ready  technology  and  until  this  point  is  realized, 
technology  transfer  will  remain  in  its  current  situation  (1993:37). 

Spann,  Adams,  and  Souder  view  the  participants  of  technology  transfer  in  a  slightly 
different  manner  (1995).  According  to  Spann  et  al.,  there  are  three  primary  participants  in 
the  technology  transfer  process:  developers,  sponsors,  and  adopters.  Developers 
“develop  and  apply  technology  under  government  funding  and  sponsorship;”  sponsors 
“fund  technology  development,  disseminate  information  about  government  technologies 
and/or  facilitate  its  transfer;”  and  adopters  “are  private  sector  firms  that  incorporate  and 
use  the  technologies  in  their  commercial  processes  or  products  as  well  as  individual  end 
users  of  the  products”  (Spann  et  al.,  1995:22).  An  example  of  a  developer  is  a  federal 
laboratory  such  as  the  Air  Force  Research  Laboratory  located  on  Wright-Patterson  Air 
Force  Base,  Ohio,  and  adopters  are  industrial  firms.  It  is  sponsors  whose  role  is  unique  to 
define.  Sponsors  perform  “go-between”  functions.  Technology  transfer  legislation 
created  Offices  of  Research  and  Technology  Applications  (ORTAs),  which  fulfill  this  role 
within  the  government  (AFMC  Handbook,  1 995  :A-2).  Once  again,  an  understanding  of 
the  participants  involved  enhances  the  technology  transfer  process. 

Schroer,  Farrington,  Messimer,  and  Thornton  describe  the  function  performed  by  the 
Huntsville/Madison  County  Chamber  of  Commerce  with  regard  to  technology  transfer. 
The  chamber  of  commerce  acts  as  a  “sponsor”  of  the  technology  transfer  process  in  the 
Huntsville,  Alabama  area.  The  mission  of  the  chamber  is  to  help  local  companies  access 
technology  at  four  federal  laboratories  located  in  north  Alabama  (Schroer  et  al.,  1995:41). 
They  accomplish  this  in  a  variety  of  ways  by  disseminating  information  about  the 
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laboratories  through  telephone  calls,  company  visits,  newsletters,  fact  sheets,  seminars  and 
workshops,  and  other  ways  (1995:40).  This  proactive  approach  has  been  successful  and 
responsible  for  much  of  the  technology  transfer  occurring  between  the  federal  laboratories 
and  the  local  business  community.  In  fact,  65%  of  participants  found  out  about 
technology  transfer  through  chamber  visits  and  10%  through  chamber  publicity  (1995:43). 
Additionally,  55%  stated  the  transfers  with  which  they  were  involved  would  increase 
revenues,  35%  would  lower  cost,  both  leading  to  an  increase  in  company  profits 
(1995:43).  Schroer  et  al.’s  research  indicates  that  providing  information  about  what  the 
federal  laboratory  can  do  to  help  industry  is  a  positive  way  of  increasing  the  frequency 
with  which  technology  transfers  occur. 

The  above  research  tells  a  story.  Both  industry  and  the  government  have  reasons  for 
pursuing  technology  transfer.  However,  the  research  also  reveals  that  technology  transfer 
is  not  pursued  as  much  as  it  might  be  if  parties  knew  about  the  other.  Communication  is 
key  in  making  technology  transfer  more  common.  Now  that  the  process  and  participants 
have  been  identified,  the  final  area  of  research  to  be  examined  is  technology  transfer 
measurement. 

Technology  Transfer  -  Measurement 

Technology  transfer  has  potential  benefits  embedded  in  its  process.  There  is  money  to 
be  made  on  both  sides:  government  may  receive  money  through  royalties  from  patents 
and  industry  can  earn  money  through  selling  products  using  the  technology  transferred. 
However,  this  simplistic  view  is  just  that,  simplistic.  In  reality,  as  the  following  research 
shows,  measuring  all  of  the  benefits  and  costs  associated  with  a  particular  transfer  of 
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technology  is  difficult.  As  of  1994,  no  measurement  method  existed  which  satisfied  Air 
Force  technology  transfer  experts  (AFMC/TTO,  1994:6). 

Carr  defines  four  measurement  models  for  technology  transfer  (1992A).  The  models, 
which  appear  below,  are  proposed  knowing  that  technology  transfer  measurement  is 
difficult  to  conceptualize,  just  as  is  technology  transfer  effectiveness: 

1 .  OUT-THE-DOOR  MODEL  -  measures  something  countable,  information  assets 
such  as  brochures,  or  in  the  case  of  the  Air  Force,  licenses  and  CRD  As.  The 
problem  with  this  model  is  the  numbers  hide  the  magnitude  and  quality  of  the 
counted  event.  (For  example,  a  single  license  may  transfer  a  groundbreaking' 
technology,  but  it  would  be  weighed  the  same  as  a  transfer  of  a  technology  with 
lesser  importance). 

2.  MARKET  IMPACT  MODEL-  desirable  since  this  is  the  goal  of  technology 
transfer.  This  model  often  uses  royalty  income  as  a  measurement,  a  reflection  of 
the  technology’s  success  within  the  market.  However,  this  model  is  not  very 
useful  because  there  is  often  a  great  deal  of  time  that  passes  between  the 
technology  transfer  beginning  and  the  receipt  of  royalties. 

3 .  POLITICAL  MODEL-  measures  items  such  as  the  number  of  reports  to 
Congress,  etc.  They  stress  successes  and  achievements  in  spending  authorized 
funds. 

4.  OPPORTUNITY  COST  MODEL-  measures  what  else  could  be  done  with  the 
money  spent  on  technology  transfer.  Not  useful  at  the  present  time  because  there 
is  no  model  in  place  to  adequately  measure  this. 

Of  the  four  suggested,  the  “opportunity  cost”  model  is  the  most  desirable,  yet  at  the  same 

time  is  the  most  out  of  reach.  It  is  easy  to  obtain  measurements  for  the  first  three  models, 

but  the  information  provided  is  not  really  meaningful.  Consider,  for  example,  the  “out- 

the-door”  model  using  the  number  of  Air  Force  CRD  As.  The  Air  Force  has  performed 

475  CRD  As  since  fiscal  year  1989  (3  in  1989;  4  in  1990;  17  in  1991;  21  in  1992;  65  in 

1993;  85  in  1994;  93  in  1995;  100  in  1996;  81  in  1997)  (Guilfoos,  1998B).  One  can  see 
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that  CRD  As  have  typically  increased  from  year  to  year,  but  vital  information  about  them  is 
unavailable  through  simply  counting  how  many  there  are.  The  fourth  model,  “opportunity 
cost,”  is  different  in  that  meaningful  information  for  decision-makers  can  result.  If  there 
was  a  way  to  measure  the  opportunity  cost  associated  with  a  particular  CRD  A,  the  R&D 
manager  would  be  able  to  make  an  educated  decision  on  whether  or  not  to  spend  their 
R&D  dollars  in  this  manner,  especially  if  technology  transfer  funds  are  “taken  out  of  hide”. 
This  points  to  the  need  for  a  return  on  investment  (ROI)  model  to  measure  the  benefits 
associated  with  the  investment  a  R&D  manager  makes  in  a  particular  technology  transfer. 
Carr’s  conclusion,  unfortunately,  is  that  technology  transfer  measurement  and  evaluation 
require  more  study  as  adequate  models  are  not  in  place  (1992A:21).  Thus,  R&D 
managers  do  not  have  a  systematic,  logical  way  to  choose  appropriate  technology  transfer 
opportunities. 

Technology  transfer  effectiveness  has  been  described  and  measured  in  a  variety  of  ways 
via  case  studies.  Winebrake  analyzed  1 16  United  States  Department  of  Energy  cases  and 
defined  success  as  the  effectiveness  of  the  transfer  process  (1992:56).  Souder  et  al. 
obtained  a  sample  of  forty  separate  programs  from  governmental  organizations  such  as 
NASA,  the  Department  of  Energy,  and  federal  laboratories;  non-governmental 
organizations  such  as  private  laboratories  and  a  research  institute;  and  university  research 
centers  (1990:7).  For  this  article,  Souder  et  al.  define  successful  technology  transfer  as 
“. . .  those  whose  technologies  were  adopted  by  more  than  three-fourths  of  the  targeted 
user  population”  (1990:7).  Adoption  occurred  when  “. . .  the  user  made  a  significant 
financial  and  emotional  commitment  to  the  routine  use  of  the  technology  for  more  than 
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two  years  and  when  the  user  expressed  satisfaction  with  the  technology”  (1990:7).  Both 
of  these  definitions  are  highly  subjective.  There  is  no  precedent  for  them.  When 
considering  Souder  et  al.’s  definition,  several  questions  come  to  mind.  What  is  the  target 
population?  What  classifies  a  significant  financial  commitment?  What  classifies  a 
significant  emotional  commitment?  Why  two  years?  What  classifies  an  expression  of 
satisfaction?  These  articles  illustrate  the  difficulty  associated  with  measuring  technology 
transfer. 

Gottinger’s  1990  article  results  mirror  those  of  Winebrake  and  Souder  et  al.  The 
article  discusses  the  social  costs  and  benefits  derived  from  technologies  that  have  spun 
off  from  the  Strategic  Defense  Initiative  (SDI).  He  identifies  five  primary  spin-off 
technologies:  the  spock  supercomputer;  high-speed  fault-tolerant  technology;  bioglass; 
laser  doppler  radar  technology;  and  the  free  electron  laser  (Gottinger,  1990:688-689). 

The  article  describes  an  “opportunity  cost”  model.  Social  benefits  are  determined  by  two 
items,  consumer  surplus  and  resource  savings  for  the  economy.  Consumer  surplus  is 
measured  by  the  dollar  savings  to  consumers  as  a  result  of  price  reduction.  The  price 
reduction  is  assumed  to  occur  because  learning  advances  associated  with  the  introduction 
of  new  technology  make  it  cheaper  to  produce  products.  Resource  savings  for  the 
economy  is  a  synonym  for  opportunity  cost  and  is  directly  related  to  consumer  surplus. 
Resource  savings  for  the  economy  is  defined  as  the  ability  of  individuals  to  spend  their 
money  elsewhere  because  they  pay  less  for  the  product  with  enhanced  technology.  In  the 
end,  Gottinger  attaches  numerical  values  to  the  five  SDI  spin-off  technologies  for 
consumer  surplus  and  resource  savings.  Gottinger  concedes  the  dollar  amounts  are 
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inaccurate  because  they  (1)  limit  benefits  experienced  to  a  specific  timeframe;  and  (2)  are 
limited  to  product  price  reductions  and  do  not  account  for  product  quality  improvement. 
While  social  benefits  are  described  above,  a  definition  for  social  costs  is  more  elusive.  In 
fact,  Gottinger  states  that  numerical  estimates  of  social  costs  are  impossible  (1990:697). 
Once  again,  technology  transfer  measurement  is  not  easy.  Gottinger  was  unable  to 
adequately  compare  social  benefits  to  social  costs,  only  a  portion  of  the  overall  benefits 
and  costs  related  to  technology  transfer. 

Guilfoos  documents  a  working  group’s  plight  to  find  a  payoff/benefits  indicator  to 
associate  with  the  transfer  of  technology  within  the  Air  Force  Materiel  Command.  The 
group  was  specifically  gathered  to  develop  a  measure  of  the  socio-economic  benefits  of 
Air  Force  technology  transfer  programs.  After  much  consideration,  the  Air  Force 
Technology  Transfer  group  of  experts  determined  they  would  identify  one  factor  to  best 
reflect  the  benefits  and  revenues  (AFMC/TTO,  1994:4).  The  group  acknowledges  from 
the  outset  three  primary  drawbacks  to  this  approach:  (1)  not  all  transfers  result  in 
revenues;  (2)  not  all  benefits  are  monetary;  and  (3)  money  may  not  be  seen  for  a  specific 
CRDA  for  a  long  period  of  time.  Thus,  the  Air  Force  technology  transfer  experts 
developed  a  measurement  tool  with  narrow  application  and  admittedly  does  not  have  the 
flexibility  to  apply  to  all  technology  transfers.  The  metric  developed  in  1994  has  not  been 
further  developed  or  pursued  since  that  time  (Guilfoos,  1998 A). 

The  above  research  indicates  a  gap  in  technology  transfer  research.  There  are  currently 
no  models  in  place  to  adequately  measure  the  return  on  investment  associated  with  a 
particular  technology  transfer.  A  look  into  common  return  on  investment  models  will 
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provide  some  foundational  information  from  which  a  model  measuring  technology  transfer 
returns  can  be  estimated. 

RQI  Models 

In  making  planning  decisions  for  investments,  there  are  several  common  methods 
utilized  by  organizations  to  determine  where  to  spend  their  money.  The  Accrual 
Accounting  Rate  of  Return  (AARR),  the  Net  Present  Value  (NPV)  and  Internal  Rate  of 
Return  (IRR)  methods  are  well  known.  Below,  each  method  is  briefly  explained  through 
simple  examples. 

Accrual  Accounting  Rate  of  Return  (AARR) 

The  accrual  accounting  rate  of  return  (AARR)  method  is  an  income  measure  divided 
by  a  measure  of  investment.  This  method  is  concerned  with  the  affect  on  operating  income 
when  an  investment  is  made.  The  associated  equation  is: 

AARR  =  Increase  in  expected  average  annual  operating  income 

Net  Initial  Investment 

A  simple  example  below  is  used  to  illustrate  how  AARR  is  used: 

A  snow  removal  company  buys  a  truck  to  dispense  sand  on  icy  roads.  The  truck  is 
purchased  for  $40,000  and  has  no  salvage  value  at  the  end  of  a  5-year  useful  life.  It  is 
expected  that  $15,000  will  be  earned  per  year  from  truck  operations.  Using  straight-line 
depreciation, 

A  Aftft  =  $1 5  000  -  f $40.000/51  =  $15.000  -  $8,000  =  $7.000  =  17.5% 
$40,000  $40,000  $40,000 

The  AARR  indicates  the  rate  at  which  a  dollar  of  investment  generates  income.  In  this 
case,  an  AARR  of  17.5%  implies  the  snow  removal  company  earns  $1.18  on  each  $1 
invested  in  the  truck.  As  one  might  guess,  the  higher  the  AARR  the  better.  However, 
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there  are  several  factors  that  often  detract  from  the  use  of  AARR.  First,  AARR  does  not 
track  cash  flows;  and  secondly,  AARR  ignores  the  time  value  of  money  (Homgren  et  al., 
1997:795).  It  is  for  these  reasons  that  the  AARR  method  is  not  utilized  in  many  instances. 

Net  Present  Value  (NPV) 

While  the  AARR  method  neglects  the  time  value  of  money,  the  Net  Present  Value 
(NPV)  adjusts  for  it.  The  NPV  method  is  a  discounted  cash-flow  method  that  calculates 
the  expected  net  monetary  gain  or  loss  from  a  project  by  discounting  all  expected  future 
cash  inflows  and  outflows  to  the  present  point  in  time,  using  a  required  rate  of  return 
(Homgren  et  al.,  1997:995).  NPV  focuses  upon  cash  inflows/outflows  with  cash  being 
invested  today  in  hopes  of  receiving  more  in  return  at  a  later  date.  The  required  rate  of 
return  represents  the  minimum  return  on  investment  acceptable;  i.e.,  the  return  the 
organization  could  obtain  by  spending  their  money  elsewhere  in  a  project  of  comparable 
risk  (1997:997).  The  required  rate  of  return  is  synonymous  to  the  discount  rate,  hurdle 
rate,  or  opportunity  cost  of  capital  (1997:783).  A  project  should  be  entered  into  if  the 
NPV  is  greater  than  or  equal  to  0,  an  indication  that  the  investment  is  profitable.  To  find 
the  NPV,  Homgren  et  al.  group  the  process  into  three  steps: 

1 .  Sketch  the  Relevant  Cash  Inflows  and  Outflows; 

2.  Choose  the  Correct  Compound  Interest  Table; 

3 .  Sum  the  Present- Value  Figures  to  Determine  the  NPV. 

The  following  example  using  the  same  snow  removal  company  illustrates  this  process: 
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The  company  buys  a  Truck:  $40,000 

Trucks  Useful  Life:  5  Years 

Recurring  Cash  flows:  $  1 5,000 

Required  Rate  of  Return:  8% 

Table  2.  NPV  With  $15K  Recurring  Cash  Flows 


Total 

Present 

Value 

Present 
Value  of  $1 
Discounted 
at  8% 

Sketch  of  Relevant  Cash  Flows 

Initial 

Investment 

($40,000) 

1.000 

$15K 

Recurring 
Cash  flows 

$13,890 

0.926 

$15K 

$12,855 

0.857 

$15K 

0.794 

$15K 

$11,025 

0.735 

$15K 

$10,215 

0.681 

$15K 

NPV 

$19,895 

As  stated  earlier,  a  project  should  be  undertaken  if  the  NPV  is  nonnegative.  With  a  NPV 


of  $19,895  at  the  required  rate  of  return  of  8%,  the  snow  removal  company  should 
definitely  purchase  the  truck.  However,  what  if  it  is  estimated  that  the  recurring  cash 
flows  are  $10,000  versus  the  previous  $15,000?  If  that  were  the  case,  the  situation 
indicated  in  the  following  chart  holds.  The  NPV  becomes  -$70,  which  is  interpreted  as 
meaning  the  company  would  actually  lose  $70  if  they  were  to  purchase  the  truck. 
Therefore,  in  this  instance,  the  truck  should  not  be  purchased. 


Table  3.  NPV  With  $10K  Recurring  Cash  Flows 


Total 

Present 

Value 

Present 
Value  of  $1 
Discounted 
at  8% 

Sketch  of  Relevant  Cash  Flows 

_ 

Initial 

Investment 

($40,000) 

1.000 

$10K 

Recurring 
Cash  flows 

$9260 

0.926 

$10K 

$8570 

0.857 

$10K 

$7940 

0.794 

$10K 

$7350 

0.735 

$10K 

$6810 

0.681 

$10K 

NPV 

$-70 

As  indicated  by  preceding  NPV  examples,  the  NPV  requires  one  to  estimate  as  to  what 


future  cash  flows  will  be.  It  is  also  limited  to  quantitative  measurements.  These  two 
pieces  of  information  make  it  somewhat  difficult  to  apply  the  NPV  method  to  all 
scenarios. 

Internal  Rate  of  Return  (IRR) 

The  internal  rate  of  return  is  the  discount  rate  at  which  the  present  value  of  expected 
cash  inflows  from  a  project  equals  the  present  value  of  expected  cash  outflows  of  the 
project  (Homgren  et  al.,  1997:993).  IRR  is  sometimes  referred  to  as  the  time-adjusted 
rate  of  return  method.  In  this  method,  a  project  is  entered  into  if  and  only  if  the  IRR  is 
greater  than  the  required  rate  of  return.  The  IRR  method  determines  the  rate  of  return  at 
which  NPV  =  0,  representing  the  point  at  which  cash  inflows  equal  cash  outflows 
(1997:785).  Once  again,  we  return  to  the  familiar  snow  removal  example  with  the 
following  information: 
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The  company  buys  a  Truck:  $39,930 

Trucks  Useful  Life:  5  Years 

Recurring  Cash  flows :  $  1 5 , 000 

Required  Rate  of  Return:  8% 

Table  4.  Internal  Rate  of  Return  With  $15K  Recurring  Cash  Flows 


Total 

Present 

Value 

Present 
Value  of  $1 
Discounted 
at  8% 

Sketch  of  Relevant  Cash  Flows 

Initial 

Investment 

— 

1.000 

$10K 

Recurring 
Cash  flows 

$9260 

0.926 

$10K 

$8570 

0.857 

$10K 

$7940 

0.794 

$10K 

$7350 

0.735 

$10K 

$6810 

0.681 

$10K 

NPV 

$0 

As  one  can  see,  the  NPV  is  0  when  the  discount  rate  is  8%.  Therefore,  the  IRR  is  8%. 


Carter  examines  the  above  accounting  methods  and  their  application  to  the  computer- 
integrated  manufacturing  (CIM)  arena.  In  order  for  CIM  to  apply  these  methods.  Carter 
states,  “an  investment’s  benefits  must  be  measurable-  in  dollars  saved  or  earned,  in  shorter 
production  schedules  and  in  higher  sales”  (1992:59).  Unfortunately  for  CIM,  “soft” 
benefits  are  important,  but  traditional  methods  are  not  flexible  enough  to  handle  them.  In 
addition  to  this  shortcoming,  Carter  lists  several  other  problems  associated  with  these 
methods: 

1 .  The  techniques  require  predictions  of  future  cash  flows,  something  rarely 
accurately  predicted. 

2.  The  models  do  not  enable  accounting  for  nonquantifiable  improvements  such  as 
an  increase  in  a  product’s  quality  and  flexibility. 
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3  Traditional  techniques  can  become  mere  rituals  with  people  simply  massaging 
their  numbers  to  support  conclusions  drawn. 

The  same  difficulties  faced  by  the  CIM  community  in  applying  traditional  ROI 
measures  are  experienced  by  Air  Force  technology  transfer  efforts.  Qualitative  benefits 
(such  as  individual  knowledge  increased  in  the  technology  transfer  process)  exist  and 
future  cash  flows  (via  royalties)  are  difficult,  if  not  impossible,  to  accurately  predict. 

If  investments  are  measured  in  the  manner  required  by  the  traditional  models,  there 
would  be  no  reason  to  perform  this  current  research.  However,  benefits  are  both 
quantitative  and  qualitative.  So  how  does  one  measure  qualitative  benefits?  The 
following  section  examines  several  approaches  to  measuring  benefits  that  do  not  directly 

translate  to  dollar  figures. 

Expansion  on  Qualitative  Measurements 

As  mentioned  above,  industry  is  experiencing  difficulty  in  measuring  qualitative 
attributes  and  has  for  some  time.  An  exception  appears  to  be  the  Natural  Gas  Industry. 
This  industry,  through  the  use  of  a  computer  algorithm  entitled  Project  Appraisal 
Methodology  (PAM),  has  been  very  successful  in  predicting  which  R&D  efforts  should  be 
undertaken,  resulting  in  positive  ROI. 

The  Gas  Research  Institute  (GRI)  was  formed  in  1978  with  its  purpose  being  to  make 
good  decisions:  to  fond  the  right  projects,  to  terminate  projects  not  meeting  goals,  to 
choose  the  right  performing  organizations  within  the  gas  industry,  and  to  pick  the  best 
path  to  commercialize  products.  At  the  time,  natural  gas  was  viewed  as  a  fuel  of  the  past. 
The  PAM  methodology,  developed  in  1975,  was  first  used  by  the  GRI  in  1978  to  provide 
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a  future  for  natural  gas.  PAM  has  led  to  good  decisions  because  it  causes  its  users  to 
“walk  the  middle  ground”  between  two  extremes:  the  “rational  actor”  model  in  which 
decision-makers  have  perfect  information  (no  uncertainty)  based  on  all  quantitative 
measures,  and  the  “gut  feel”  approach,  solely  based  on  guesses  (Burnett  et  al., 

1993B:  1022).  PAM  accomplishes  this  middle  ground  by  answering  7  basic  questions 
(1993B:  1024-1026): 

1.  What  will  the  project  accomplish? 

2.  What  are  the  potential  markets? 

3.  What  impact  will  each  project  have  on  each  market? 

4.  What  are  the  benefits  and  costs  which  result  from  the  projected  market  impacts? 

5.  What  is  the  probability  that  the  calculated  benefits  will  be  achieved? 

6.  What  benefits  (if  any)  will  come  about  if  GRI  doesn’t  fund  the  project? 

7.  What  is  the  expected  impact  of  the  GRI  R&D  project  at  a  given  funding  level? 

This  set  of  questions  evaluates  a  potential  project  from  its  conceptual  stage  through  its 
completion,  including  what  the  organization  may  lose  if  it  does  not  conduct  the  project. 
The  final  question  is  akin  to  sensitivity  analysis,  in  that  GRI  looks  into  the  return  on 
investment  expected  given  certain  levels  of  investment.  The  methodology  has  been 
successful  for  the  natural  gas  industry,  with  the  primary  results  listed  below: 

1 .  A  HIGH  SUCCESS  RATE  -  PAM  is  an  excellent  prediction  tool. 

GRI  has  had  132  successful  project  commercializations  and  has  a  30%  success 
rate,  two  times  the  industry  average.  “GRI’s  record  of  success  exceeds  that  of  any 
other  R&D  funding  organization  that  has  documented  its  results  and  even 
substantially  exceeds  the  reported  average  of  US  industrial  firms”  (Burnett  et  al., 
1993A:50). 

2.  A  HIGH  BENEFIT-TO-COST  RATIO  AND  A  HIGH  RATE  OF  RETURN 
GRI  is  more  than  breaking  even  on  their  expenditures  (1993 A:51). 

3.  INCREASED  GAS  AVAILABILITY  AT  LOWER  PRICES 

GRI  led  a  major  program  to  discover  natural  gas  sources  like  coal  seams  and  sand, 
with  technology  they  helped  invent  (1993 A:52). 
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4.  INCREASED  GAS  MARKET  SHARE 

More  uses  for  natural  gas  have  been  developed  by  GRI,  broadening  the  consumer 
base,  resulting  in  more  sales  (1993 A:52). 

5.  COST  SAVINGS  IN  TRANSMISSION  AND  DISTRIBUTION 

GRI  has  developed  over  30  technologies  which  have  lessened  the  costs  associated 
with  getting  the  oil  from  its  natural  holding  place  into  the  hands  of  customers 
(1993 A:54). 

PAM  has  been  highly  effective  in  achieving  the  above  results  because  it  strikes  a 
workable  balance  between  judgment  and  formal  methodology  (Burnett  et  al,  1993 A:50). 
This  flexibility  enables  quantitative  measures  to  be  considered  along  with  experts’ 
thoughts  concerning  nonquantifiable  measures.  In  addition  to  the  above  benefits,  the  gas 
industry’s  gains  have  passed  to  related  industries  such  as  the  steel,  glass  and  aluminum 
industries  (Burnett  et  al.,  1993 A: 53-54).  Although  PAM  appears  to  be  applicable  to 
estimating  ROI  for  Air  Force  Technology  Transfer,  the  computer  program  is  not  pursued 
due  to  its  proprietary  nature.  Instead,  this  effort  continues  by  exploring  a  similar  approach 
to  PAM  which  is  found  in  the  non-proprietary  methodology  introduced  by  Canada  and 
Sullivan. 

Canada  and  Sullivan  document  how  to  measure  project  return  on  investment  without 
the  use  of  a  computer  program  in  their  book.  Economic  and  Multiattribute  Evaluation  of 
Advanced  Manufacturing  Systems.  Just  as  Burnett  et  al.  describe  a  balance  between 
quantitative  and  qualitative  benefits  of  Natural  Gas  R&D,  so  do  Canada  and  Sullivan  for 
Manufacturing  Technology  (1989).  Chapter  8  of  their  book  introduces  multiattribute 
decision  analysis  and  basic  techniques  (1989:21 1).  The  elementary  steps  of  the  process 
are  to:  (1)  list  all  attributes,  ensuring  they  are  independent  of  one  another,  and  (2)  weigh 
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the  attributes  according  to  their  importance.  There  are  many  methodologies  associated 
with  both  steps.  A  listing  of  attributes  may  be  all  inclusive  or  on  the  other  hand  might  be 
based  on  the  principle  of  parsimony,  listing  those  factors  contributing  the  most  and 
ignoring  those  having  only  minor  contributions  (1989:212-213).  Weighting  of  the 
attributes  may  be  done  by  the  equal  weight,  rank  sum  weight,  or  rank  reciprocal  weight 
methods,  to  name  a  few  (1989:226).  This  simplistic  method  allows  flexibility  in  analysis 
and  requires  only  basic  information.  These  characteristics  make  their  methods  desirable  to 
the  layperson  attempting  to  determine  whether  or  not  to  embark  upon  a  specific  project. 

Canada  and  Sullivan’s  multiattribute  decision  analysis  methodology  is  an  excellent  tool 
to  use  in  analyzing  the  return  on  investment  for  Air  Force  technology  transfer.  Canada 
and  Sullivan  provide  a  framework  in  which  to  gather  information  concerning  both  the 
costs  and  benefits  of  technology  transfer.  Next,  the  methodology  is  geared  toward  expert 
judgment.  This  means  that  the  people  with  the  most  knowledge  concerning  technology 
transfer  are  the  ones  providing  input  concerning  costs/benefits.  Also,  gathering  of 
information  is  relatively  easy  and  is  not  time-consuming  (NOTE:  Attachments  A  &  B 
were  used  to  gather  the  information  for  this  research).  The  bottom  line  is  that  the 
methodology  is  simple  to  use  and  allows  for  user  flexibility  (e.g.,  exhaustive  list  of  costs  or 
inclusion  of  “most  important”  costs;  and  weighting  attributes  according  to  equal  weight, 
rank  sum  weight,  or  rank  reciprocal  weight).  For  these  two  primary  reasons  (simplicity 
and  flexibility),  the  Canada  and  Sullivan  multiattribute  decision  analysis  methodology  will 
be  the  primary  tool  utilized  to  fulfill  the  objectives  of  this  research  effort. 


39 


Previous  Research  on  Which  to  Build 

While  there  currently  is  no  model  in  place  to  measure  ROI  for  Air  Force  technology 
transfer,  some  of  the  parts  needed  to  build  that  model  are  present.  In  order  to  measure  a 
return  on  investment,  there  are  two  elementary  parts:  the  investment  (direct  and  indirect 
costs)  and  the  returns  (quantitative  and  qualitative  benefits).  Below  are  four  past  research 
efforts  and  how  they  contribute  to  the  development  of  an  ROI  model. 

Boyd  determined  the  direct  costs  of  technology  transfers  in  his  research  (1996).  His 
research  effort  examined  eight  transfer  projects  at  an  Air  Force  Laboratory  using  Activity- 
Based  Costing  (ABC).  His  research  allocates  the  direct  costs  according  to  the  six 
technology  transfer  steps  defined  by  the  AFMC  Handbook:  (1)  Strategy,  (2)  Technology 
Identification,  (3)  Marketing,  (4)  Vehicle  Identification,  (5)  Transfer,  and  (6)  Post-transfer 
administration.  Boyd  found  that  no  direct  costs  were  allocated  to  steps  1-3,  step  4 
consumed  3%  of  resources,  step  6  consumed  5%  of  resources,  and  the  remaining  92%  of 
technology  transfer  resources  were  dedicated  to  step  5,  the  actual  transfer  (Boyd, 

1996:ix).  Of  that  92%,  80%  of  those  resources  were  human  resource  related  (Boyd, 
1996:ix).  Examples  of  direct  costs  included  equipment,  travel,  and  technicians,  but  the 
majority  was  associated  with  the  scientists  and  engineers  (Boyd,  1996:4-49).  It  is  Boyd’s 
conclusion  that  having  92%  of  the  resources  spent  in  the  transfer  stage  is  good  because 
that  is  the  “rubber  meeting  the  road”  stage  (1996:5-4). 

Van  Egeren  examined  the  indirect  costs  of  technology  transfer  in  his  recent  research 
effort  (1997).  Van  Egeren’s  objective  was  to  determine  the  indirect  costs  of  performing 
technology  transfer  (1997:ix).  The  research  attempted  to  accomplish  this  by  identifying 


the  resources  consumed  and  activities  performed  by  ORTA  organizations.  Van  Egeren’s 
study  covered  federal  laboratories,  Air  Logistics  Centers,  and  test  centers,  where  he  found 
significant  differences  between  the  laboratories,  ALCs,  and  test  centers,  and  also  within 
them  (1997:74).  His  results  were  once  again  measured  using  the  6  steps  of  the  technology 
transfer  process  as  a  backdrop.  Unlike  Boyd,  Van  Egeren  found  that  the  first  major  step, 
strategy,  consumed  the  most  resources  (1997:90).  In  fact,  strategy  consumed  over  25% 
of  ORTA  resources.  This  makes  sense  because  strategy  is  managerial  in  nature  and  falls 
right  into  the  roles  ORTAs  were  created  for,  to  manage.  Marketing,  the  third  task,  used 
over  one  fifth  of  the  total  ORTA  resources.  In  the  end,  Van  Egeren  was  able  to  assign  an 
average  overhead  cost  to  each  CRD  A,  based  upon  the  CRD  As  implementers,  whether  it 
be  an  Air  Logistics  Center  or  a  federal  laboratory,  such  as  Wright  Laboratory  (1997:89). 
The  specific  ORTA  cost  per  CRD  A  at  Wright  Laboratory  was  $12,3 18. 

While  Boyd  and  Van  Egeren  explored  technology  transfer  costs  (direct  and  indirect, 
respectively),  Braun  attempted  to  define  the  tangible  (quantitative)  and  intangible 
(qualitative)  benefits  associated  with  technology  transfer  (1996).  Braun’s  study  looked  at 
information  on  past  technology  transfers,  based  on  questionnaires  filled  out  by  those 
performing  the  technology  transfer.  Braun  found  many  quantifiable  and  non-quantifiable 
benefits.  Quantifiable  benefits  included  royalty  payments,  cash  reimbursements,  work 
savings,  resource  savings,  productivity  increases,  effective  resource  utilization,  and  data 
exchange  savings  (Braun,  1996:44).  Non-quantifiable  benefits  included  public  relations 
and  goodwill,  enhanced  morale,  technology  dissemination,  technology  improvements,  use 
of  additional  technologies,  enhanced  social  wealth,  follow-on  CRD  A  activities,  and 
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improvements  in  the  transfer  process  (1996:44).  Braun  found  that  qualitative  benefits  are 
important,  but  they  are  difficult  to  quantify  (1996:65).  Problems  exist  because  there  is  no 
generally  accepted  measure  available  and  there  is  a  concern  for  bias  if  an  expert  were 
given  free  reign  to  determine  the  measurements  on  their  own  (1996:65).  Nevertheless, 
Braun  succeeded  in  outlining  the  attributes  considered  important  to  experts,  tangible  and 
intangible. 

A  final  technology  transfer  research  effort  of  interest  was  completed  by  Davis  (1997). 
Davis  examined  Cooperative  Research  &  Development  Agreements  (CRD As)  and  the 
elements  of  them  that  are  most  important  to  the  federal  government  receiving  benefits 
from  technology  transfer  projects.  From  this  data,  CRDA  elements  most  associated  with 
higher  benefits  to  the  government  were:  quantified  manpower  requirements,  commercial 
partner’s  ability  to  commercialize  the  specific  technology,  CRDA  technology  market 
information,  quantified  copyright  royalty  rates,  and  quantified  sales  royalty  rates  (Davis, 
1997:vi).  Davis’  analysis  was  performed  using  69  CRD  As,  a  major  compilation  of  data. 
This  data  will  be  utilized  as  part  of  the  ex  post  facto  study  performed  in  the  following 
chapters. 

Technology  Transfer  and  ROI  Models 

It  has  been  shown  there  is  no  one  specific  model  that  is  directly  applicable  to  measure 
ROI  for  Air  Force  technology  transfer.  This  has  contributed  to  the  lack  of  participation  in 
technology  transfer  above  mandated  levels.  Traditional  methods  have  drawbacks  and 
current  theoretical  models  do  not  provide  for  direct  application.  Because  this  situation 
exists,  in  this  research,  an  effort  is  made  to  create  a  ROI  model  by  combining  pieces  of  the 
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models  above.  This  research  will  examine  both  qualitative  and  quantitative  measures.  As 
Canada  states,  “sensitivity  analysis  is  needed  because  this  isn’t  an  exact  science” 
(1989:237).  Therefore,  this  model  will  provide  for  the  opportunity  to  perform  this  type  of 
analysis.  Upon  completion  of  this  research  effort,  it  is  desired  that  a  ROI  model  will  be 
developed,  capable  of  being  used  by  Air  Force  decision-makers  affiliated  with  the 
technology  transfer  process. 

The  Air  Force  technology  transfer  process  should  take  lessons  from  previous  private 
sector  success.  As  indicated  earlier,  the  Gas  Research  Institute  (GRI)  greatly  benefited 
from  the  PAM  methodology.  Recall  that  three  primary  benefits  of  using  PAM  were: 

1 .  PAM  was  excellent  prediction  tool  -  appropriate  technologies  were  developed. 

2.  PAM  improved  GRI’s  benefit-to-cost  ratio  and  rate  of  return  -  higher  return  on 
investment. 

3.  Technologies  were  developed  which  lessened  other  costs  -  unforeseen  benefits 
resulted. 

The  model  developed  in  this  research  effort  will  contribute  to  Air  Force  decision-makers 
being  able  to  strategically  achieve  the  above  three  results.  While  the  model  developed  will 
not  be  a  “silver  bullet”,  it  is  a  step  toward  making  transfer  decisions  less  “gut  feel”  and 
more  structured  in  nature. 

Research  Question 

Which  technology  transfer  opportunities  should  the  USAF  pursue  in  order  to 
receive  the  greatest  ROI? 
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Summary 


This  chapter  focused  on  examining  research  in  existence  on  technology  transfer  as  well 
as  return  on  investment  models  (ROI).  The  chapter  reviewed  the  technology  transfer 
legislation  in  place,  as  well  as  literature  concerning  the  transfer  process,  its  participants, 
and  measurement.  The  chapter  reviewed  ROI  models  and  their  makeup.  It  was  also 
shown  there  is  currently  no  ROI  model  in  place  to  determine  which  technology  transfer 
projects  should  be  entered  into.  Burnett  et  al.  lay  the  theoretical  foundations  for 
developing  a  return  on  investment  model  by  their  description  of  the  PAM  methodology. 
Canada  and  Sullivan  provide  the  means  to  take  Burnett  et  al.'s  theory  and  practically 
develop  a  model  for  measuring  Air  Force  Technology  Transfer  ROI.  Chapter  III  describes 
the  methodology  associated  with  developing  a  ROI  model  suitable  for  use  by  Air  Force 
technology  transfer  decision-makers. 
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PI.  Methodology 


Introduction 

The  purpose  of  this  chapter  is  to  develop  a  model  that  will  enable  the  determination  of 
return  on  investment  (ROI)  of  individual  CRD  As.  This  model  will  be  capable  of 
evaluating  past  CRD  As  and  maintain  the  flexibility  to  perform  sensitivity  analysis.  As 
shown  by  the  success  of  the  PAM  methodology,  an  ability  to  measure  both  subjective  and 
objective  benefits  is  of  paramount  importance  in  accurately  determining  ROI.  It  was  also 
indicated  in  the  literature  review  that  Canada  and  Sullivan’s  multiattribute  analysis 
methodology  is  capable  of  capturing  the  same  kinds  of  information  that  PAM  does. 
Therefore,  Canada  and  Sullivan’s  methodology  is  the  primary  basis  for  the  model 
proposed  within  this  chapter. 

Data  Collection 

There  are  two  primary  sources  of  data  utilized  by  this  research:  previous  work  on  the 
subject  and  data  gathered  specifically  for  this  research  by  the  author.  It  is  important  to 
note  that  all  data  discussed  in  this  research  is  limited  in  scope  to  technology  transfer 
efforts  of  the  portion  of  Air  Force  Research  Laboratory  (AFRL)  formerly  known  as 
Wright  Laboratories  (WL).  This  will  provide  a  baseline  from  which  to  develop  a  model, 
which  upon  completion,  can  be  calibrated  and  used  by  other  organizations.  The  WL 
portion  of  AFRL  was  the  target  of  choice  due  to  its  proximity  to  the  Air  Force  Institute  of 
Technology  (AFIT)  and  the  working  relationship  between  its  technology  transfer  experts 
and  those  of  AFIT.  Insight  into  the  gathering  of  each  data  set  is  presented  next. 
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Three  sources  of  data  already  available  from  previous  research  efforts  are  that  from 
Boyd,  Van  Egeren,  and  Davis.  Boyd  and  Van  Egeren  provide  cost  information,  whereas 
Davis  examines  benefits.  Placing  them  together,  we  have  the  key  elements  (i.e.,  costs  and 
benefits)  to  determine  ROI. 

Boyd  and  Van  Egeren  collectively  address  the  costs  associated  with  technology 
transfer.  In  his  1996  research  effort,  Boyd  determined  the  direct  costs  of  eight  technology 
transfer  projects  at  Wright  Laboratory  using  Activity-Based  Costing  (ABC).  Boyd 
determined  the  total  direct  cost  of  8  CRD  As  to  be  $370,237,  an  average  of  $46,280  each 
(Boyd,  1996:4-46).  Van  Egeren  examined  the  indirect  costs  associated  with  Wright 
Laboratory  CRD  As  in  his  1997  thesis.  Indirect  costs  were  approximated  by  the  ORTA 
cost  per  CRD  A.  His  research  determined  the  indirect  cost  of  82  CRD  As  to  be  $12,318. 
Boyd  and  Van  Egeren  developed  average  values  for  CRDA  direct  and  indirect  costs. 
Because  their  results  produced  a  single  value  for  each  cost  category,  the  numbers  cannot 
be  applied  to  the  analysis  in  Chapter  IV  because  they  are  tantamount  to  “wash  costs”  (they 
are  the  same  for  all  alternatives).  However,  their  research  is  important  in  that  it  shows 
that  it  is  possible  to  calculate  the  direct  and  indirect  costs  associated  with  CRD  As. 

Davis’  results  are  relied  on  more  heavily  than  those  of  Boyd  and  Van  Egeren  for  this 
research  project.  In  1997,  he  examined  69  separate  Wright  Laboratory  CRDAs  to 
determine  the  objective  and  subjective  benefits  associated  with  each  CRDA. 

Objective  benefits,  in  general,  are  benefits  that  are  tangible  (can  be  measured/counted). 
For  technology  transfer,  Davis  identified  the  following  objective  benefits:  royalty 
payments,  cash  reimbursements,  license  fees,  work  avoidance,  data  received,  software 


46 


received,  and  hardware  received.  An  objective  benefit  exists  (in  each  category)  if  the 
following  questions  are  answered  in  the  affirmative. 

1 .  Royalties  -  Did  the  government  receive  any  kind  of  royalty  payment  as  a  direct  result 
of  this  CRDA? 

2.  Cash  reimbursement  -  Did  the  government  receive  any  kind  of  cash  reimbursement  for 
services  performed  by  the  laboratory  as  a  direct  result  of  this  CRDA? 

3 .  License  fees  -  Did  the  government  receive  any  kind  of  license  fees  as  a  direct  result  of 
this  CRDA? 

4.  Work  avoidance  -  Did  the  government  realize  any  financial  benefits  that  are  not 
directly  payable  as  cash  as  a  direct  result  of  this  CRDA:  specifically  work  avoidance? 

5.  Data  received  -  Did  the  government  realize  any  financial  benefits  that  are  not  directly 
payable  as  cash  as  a  direct  result  of  this  CRDA;  specifically  data  received? 

6.  Software  received  -  Did  the  government  realize  any  financial  benefits  that  are  not 
directly  payable  as  cash  as  a  direct  result  of  this  CRDA:  specifically  software 
received? 

7.  Hardware  received  -  Did  the  government  realize  any  financial  benefits  that  are  not 
directly  payable  as  cash  as  a  direct  result  of  this  CRDA:  specifically  hardware 
received? 

If  objective  benefits  existed  for  a  CRDA  in  any/all  of  the  seven  categories,  dollar  estimates 
were  provided  to  indicate  how  much  of  each  specific  benefit  a  CRDA  contained. 

Subjective  benefits  identified  by  Davis  were:  productivity,  resource  utilization, 
management,  image,  morale,  and  technical  capability.  Subjective  benefits  were  defined  to 
exist  (in  each  individual  category)  if  the  answer  to  the  following  questions  was  “yes”: 

1.  Productivity  -  Did  this  CRDA  result  in  productivity  increases  in  the  laboratory  (e.g.,  is 
the  lab  more  efficient  in  its  technology  development)? 

2.  Resource  utilization  -  Did  this  CRDA  result  in  better  utilization  of  specialized 
laboratory  resources  (e.g.,  more  use  of  a  laboratory  test  facility)? 
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3.  Management  -  Did  this  CRDA  result  in  improved  management  and  business  practices 
in  the  laboratory  (e.g.,  improved  skills  and  techniques  through  interaction  with  the 
partner)? 

4.  Image  -  Did  this  CRDA  result  in  improved  laboratory  image? 

5.  Morale-  Did  this  CRDA  result  in  improved  morale  in  the  laboratory? 

6.  Technical  Capability-  Did  this  CRDA  improve  lab  personnel  technical  capability? 
Subjective  benefits  are  intangible;  that  is,  they  cannot  be  measured  in  countable  terms, 
such  as  dollars.  Once  it  was  determined  that  a  CRDA  contained  such  a  benefit,  a  5-point 
Likert  scale  (ranging  from  1  being  “not  important”  to  5  being  “extremely  important”)  was 
provided  to  which  experts  responded  by  giving  their  opinion  as  to  the  importance  of  an 
individual  subjective  benefit.  The  point  of  this  was,  if  the  benefit  existed  but  was  not 
considered  to  be  important  by  that  expert,  other  alternatives  with  more  desirable  benefits 
should  be  pursued. 

In  addition  to  the  previous  research  efforts,  the  author  gathered  new  data  through  the 
creation  and  distribution  of  a  survey.  The  purpose  of  the  survey  (Appendix  B)  was  to 
gather: 

1 .  A  measurement  of  the  comparative  importance  of  objective  to  subjective  benefits; 

2.  A  measurement  of  the  importance  of  individual  objective  benefits  to  other 
objective  benefits; 

3 .  A  measurement  of  the  importance  of  individual  subjective  benefits  to  other 
subjective  benefits. 

The  data  obtained  from  this  survey  provide  the  final  piece  of  necessary  information  to 
enable  usage  of  the  Canada/Sullivan  multiattribute  decision  analysis  methodology. 
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Methodology  Applicability 


The  Canada/Sullivan  multiattribute  decision  analysis  methodology  is  located  in  Chapter 
Eight  of  their  textbook  entitled,  Economic  and  Multiattribute  Evaluation  of  Advanced 
Manufacturing  Systems.  Those  familiar  with  Air  Force  technology  transfer  may  wonder  if 
a  methodology  based  upon  advanced  manufacturing  systems  is  appropriate/applicable  to 
evaluate  technology  transfer  projects.  Wright  Laboratory  is  involved  in  the  transfer  of 
various  technologies  in  addition  to  manufacturing  (e.g.  avionics).  While  this  is  true, 
Canada  and  Sullivan  allay  the  fear  of  inapplicability  by  clearly  stating  their  methodology  is 
not  tied  to  manufacturing  systems  alone.  There  is  a  fundamental  link  between  advanced 
manufacturing  systems  and  technology  transfer:  both  contain  objective  and  subjective 
benefits.  The  identification  of  these  benefits  and  then  combining  them  in  such  a  manner  as 
to  provide  a  rank-order  of  the  potential  alternate  actions  is  the  foundation  of  the 
methodology.  Of  secondary  importance  is  the  fact  the  authors  indicate  in  the  preface  of 
their  book  that  Chapter  Eight  (where  the  methodology  is  located)  can  be  utilized 
independently  from  the  rest  of  the  book  (Canada  and  Sullivan,  1989:xiv).  The  purpose  of 
their  multiattribute  decision  analysis  methodology  is  to  combat  two  common  problems 
they  have  identified  in  highly  complex  decisions.  These  problems  are  quoted  below: 

A.  Inclinations  to  let  feelings  overcome  logic; 

B.  Tendencies  to  take  shortcuts  that  violate  formal  logic/rationality 

(Canada  and  Sullivan,  1989:11) 

Canada  and  Sullivan’s  intentions  were  to  attack  these  two  areas.  It  is  the  purpose  of 
this  research  to  make  this  applicable  to  Air  Force  technology  transfer  by  providing 
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decision-makers  a  “tool”  to  incorporate  logic  and  “feel”  together.  The  goal  of  this 
research  is  in  line  with  the  goal  of  Canada/Sullivan,  to  simplify  complex  decisions  through 
the  use  of  logic.  This  methodology  provides  the  framework  to  accomplish  this. 
Methodology  Specifics 

The  ultimate  goal  of  the  methodology  is  to  determine  a  relative  ranking  among 
prospective  technology  transfer  projects.  This  comparison  is  accomplished  by  a  weighted 
evaluation  of  each.  The  weighted  evaluation  (WEk)  is  defined  in  Equation  1 : 


WEk  -  (a)  *  (OMk)  +  (1  -  a)  *  (SMk)  (1) 

where: 

OMk  -  the  objective  value  for  alternative  k 
SMk  -  the  subjective  value  for  alternative  k 
a  -  relative  importance  weighting  for  objective  values  versus  subjective  values, 
and  (0  <  a  <  1) 

The  higher  the  weighted  evaluation  an  alternative  has,  the  more  desirable  the  alternative. 
Theoretically,  the  alternative  with  the  highest  WEk  is  the  one  that  should  be  pursued.  The 
weighted  evaluation  for  a  particular  alternative  is  composed  of  the  objective  value  for  the 
alternative,  the  subjective  value  for  the  alternative,  and  the  relative  importance  of  the 
objective  to  subjective  value.  The  remainder  of  this  section  applies  these  concepts  to  Air 
Force  technology  transfer  concepts  and  terminology  through  the  use  of  simple, 
hypothetical  data.  (NOTE:  Chapter  IV  will  contain  the  actual  data  collected  for  this 

research  project). 


50 


For  Air  Force  technology  transfer,  the  objective  measure  for  each  alternative  is 


composed  of: 

BENEFITS* 

Royalty  Payments 
Cash  Reimbursements 
License  Fees 
Work  Avoidance 
Data  Received 
Software  Received 
Hardware  Received 

(*  Recall,  direct  costs  were  found  by  Boyd,  indirect  costs  by  Van  Egeren,  and  benefits  by 
Davis). 

The  objective  measurement  for  a  particular  alternative  (Canada  and  Sullivan, 
1989:233)  is  found  by  using  Equation  2: 


COSTS* 
Direct  Costs 
Indirect  Costs 


OMk  = 


OFPk 
I lOFR 


(2) 


where: 

OMk  -  the  overall  objective  measure  for  alternative  k 

OFPk  -  the  objective  measure  for  alternative  k  (which  is  divided  by  the  sum  of  the 
objective  measures  for  all  alternatives,  1  to  K,  to  obtain  OMk). 

K  -  total  number  of  alternatives 

In  simple  terms,  this  equation  means  the  objective  measurement  value  for  alternative  k  is 
equal  to  the  total  value  of  its  objective  measurements  divided  by  the  total  value  of  the 
objective  measurements  of  all  possible  alternatives  summed  together. 

For  Air  Force  technology  transfer,  the  subjective  measure  for  each  alternative  is 
composed  of  the  following  potential  benefits: 
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BENEFITS* 


Productivity 

Resource  Utilization 

Management 

Image 

Morale 

Technical  Capability 

(*Recall,  subjective  benefits  were  found  by  Davis). 

Furthermore,  the  subjective  measurement  for  alternative  k  (Canada  and  Sullivan, 
1989:235)  is  found  by  using  Equation  3: 


ZN 

j  (subjective  attribute  weight;)  *  (subjective  evaluation  rating  *)  (3) 


where: 


SMk  -  the  overall  subjective  measure  for  alternative  k 
Subjective  attribute  weight;  -  the  measure  of  overall  importance  of  a  particular 

subjective  attribute  in  relation  to  all  other  subjective 
attributes. 

Subjective  evaluation  rating*  -  the  measure  of  how  much  of  attribute  i  alternative  k 

contains. 

N  -  total  number  of  subjective  attributes. 


The  subjective  attribute  weight  (also  known  as  the  “unitized  weight”)  indicates  the 
importance  of  an  individual  subjective  benefit  in  relation  to  the  other  subjective  benefits. 
If  one  benefit  were  “1/2  the  importance”  of  all  benefits,  that  benefit  would  have  a 
subjective  attribute  weight  of  0.50.  However,  this  example  will  contain  the  simplest 
relationship  and  assume  all  six  subjective  benefits  are  equally  important.  This  implies  a 
subjective  attribute  weight  of  0.1667  for  each  benefit  since  the  sum  total  must  be  1.0. 
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The  subjective  evaluation  rating  for  each  are  captured  by  a  Likert  scale.  The  following 
Likert  scale  is  used  to  “measure”  subjective  benefits  associated  with  Air  Force  technology 
transfer  with  the  highest  value  being  the  most  desirable: 


Table  5.  Likert  Scale  for  Subjective  Benefits 


VALUE 

1 

2 

3 

4 

5 

INTERPRETATION 

NOT 

IMPORTANT 

SLIGHTLY 

IMPORTANT 

MODERATELY 

IMPORTANT 

VERY 

IMPORTANT 

EXTREMELY 

IMPORTANT 

Putting  this  information  together,  we  are  able  to  determine  the  subjective  measurement 
values  for  each  individual  CRD  A. 


The  relative  importance  weighting  for  OMk  is  a  subjective  measurement  typically 
determined  by  the  expert  utilizing  the  methodology.  In  this  hypothetical  example,  let  a  be 
0.5  (i.e.,  objective  and  subjective  measures  are  deemed  equally  important). 

At  this  point,  an  example  is  appropriate: 

EXAMPLE  3-1; 

Given:  3  separate  CRD  As  with  hypothetical  inputs: 

STEP  1:  Determine  the  OMk  for  each  CRD  A. 

Recall  Equation  2  is  used  to  determine  OMk. 
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Table  6.  Determining  Objective  Measurement  Values 


OBJECTIVE  MEASUREMENTS: 

COSTS 

CRDA  1 

CRDA  2 

CRDA  3 

CATEGORY 

TOTALS 

Direct  Costs 

($25,000) 

($20,000) 

($15,000) 

($60,000) 

Indirect  Costs 

($40,000) 

($40,000) 

($40,000) 

($120,000) 

BENEFITS  _ 

Royalty  Payments 

$5,000 

$6,000 

$7,000 

$18,000 

Cash  Reimbursements 

$40,000 

$30,000 

$50,000 

$120,000 

License  Fees 

$1 ,000 

$4,000 

$7,000 

$12,000 

Work  Avoidance 

$70,000 

$45,000 

$60,000 

$175,000 

Data  Received 

$22,000 

$2,000 

$0 

$24,000 

Software  Received 

$1,000 

$0 

$3,000 

$4,000 

Hardware  Received 

$2,000 

$2,300 

$2,600 

$6,900 

CRDA 

TOTALS 

$76,000 

$29,300 

$74,600 

$179,900 

O 

2 

II 

0.4225 

0.1629 

0.4147 

As  shown  in  Table  6,  the  objective  measurements  for  CRD  As  1, 2,  and  3  are  0.4225 
($76,000/$  179,900),  0.1629  ($29,300/$179,900),  and  0.4147  ($74,600/$  179,900), 

respectively.  (NOTE:  ^  0Mk=1)-  Relatively  speaking,  this  implies  CRDA1  is 


superior  on  the  basis  of  objective  measurements  alone  with  CRD  A3  a  close  second.  Next, 
subjective  measures  are  examined. 

STEP  2:  Determine  the  subjective  measurement  (SMk)  for  each  CRD  A. 


Recall,  Equation  3  is  used  to  determine  SMk: 

The  step-by-step  procedure  demonstrated  in  Table  7  is  described  below: 

1 .  Input  CRDA  benefit  values  for  each  subjective  benefit. 

2.  Sum  those  values  into  the  “category  totals”  column. 
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3 .  Determine  the  fractional  value  of  the  individual  CRD  As  benefit  value  to  the  entire 
“category  total”  and  insert  into  the  second  portion  of  the  table,  (e.g.,  CRDA1  has  a 
value  of  5  for  subjective  benefit  “productivity”.  The  category  total  is  15.  Thus, 
the  fractional  value  is  0.33). 

4.  Find  the  SM  for  each  CRDA  by  multiplying  the  unitized  weight  for  each  subjective 
benefit  by  the  fractional  value  determined  in  step  3  and  summing  these  up  for  the 
individual  CRDA.  For  example,  the  subjective  measurement  value  for  CRDA  1  is: 

SMcrdai=  (0.167*0.3333)  +  (0.167*0.2727)  +  (0.167*0.3636)  +  (0.167*0.3333) 

+(0.167*0.3000)  +  (0.167*0.3333)  =  0.3227. 

Table  7  shows  the  results  for  all  three  CRD  As  utilizing  the  preceding  steps.  The  results 
indicate  that,  based  on  subjective  measurements  alone,  CRD  A3  is  superior  to  both 
CRDA1  and  CRDA2. 


Table  7.  Determining  Subjective  Measurement  Values 


SUBJECTIVE 

MEASUREMENTS: 

UNITIZED 

WEIGHT  BENEFITS 

CRDA  1 

CRDA  2 

i 

CRDA  3 

CATEGORY 

TOTALS 

0.167 

Productivity 

5 

5 

5 

15 

0.167 

Resource  Utilization 

3 

4 

4 

11 

0.167 

Management 

4 

3 

4 

1  1 

0.167 

Image 

4 

4 

4 

12 

0.167 

Morale 

3 

2 

5 

10 

0.167 

Technical  Capability 

4 

3 

5 

12 

UNITIZED 

WEIGHT 

BENEFITS 

CRDA  1 

CRDA  2 

CRDA  3 

TOTALS 

0.167 

Productivity 

0.3333 

0.3333 

0.3333 

1 

0.167 

Resource  Utilization 

0.2727 

0.3636 

0.3636 

1 

0.167 

Management 

0.3636 

0.2727 

0.3636 

1 

0.167 

Image 

0.3333 

0.3333 

0.3333 

1 

0.167 

Morale 

0.3000 

0.2000 

0.5000 

1 

0.167 

Technical  Capability 

0.3333 

0.2500 

0.4167 

1 

SMk  = 

0.3227 

0.2922 

0.3851 

1 
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To  this  point,  we  have  seen  that  CRDA1  is  superior  on  objective  measures  alone. 

Also,  we  have  noted  that  CRDA3  is  superior  on  subjective  measures  alone.  The  final 
factor  needed  in  the  multiattribute  decision  analysis  methodology  to  determine  the 
weighted  evaluation  for  each  alternative  is  a,  the  relative  importance  weighting  of  OMk 
Recall,  for  this  example,  a  is  assumed  to  be  0.50. 

STEP  3:  Determine  the  weighted  evaluations  for  each  alternative  (WEk). 

Recall,  WEk  =  (a)  *  (OMk)  +  (1  -  a)  *  (SMk) 

Applying  this  formula,  we  have: 

WEcrdai  =  (0.5)  *  (0.4225)  +  (0.5)  *  (0.3227)  =  0.3726 

WEcrda2  =  (0.5)  *  (0.1629)  +  (0.5)  *  (0.2922)  =  0.2275 

WEcrdas  =  (0.5)  *  (0.4147)  +  (0.5)  *  (0.3851)  =  0.3999 

Thus,  CRD A3  is  the  alternative  of  choice  based  upon  weighted  evaluation  values.  This 
highlights  the  fact  that,  when  objective  and  subjective  benefits  are  considered  together,  the 
alternative  of  choice  may  be  different  than  if  only  objective  benefits  were  considered. 
STEP  4:  Perform  Sensitivity  Analysis  (if  applicable/desired). 

There  are  many  possible  approaches  to  sensitivity  analysis  when  utilizing  this 
methodology.  One  example  is  to  vary  the  relative  importance  of  objective  to  subjective 
benefits  (a)  from  0  to  1 .  The  results  of  performing  this  sensitivity  analysis  follow  in 
Figure  2.  The  chart  reveals  that  CRD  A3  is  the  optimal  alternative  where  a  ranges 
between  0  and  0.9. 
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However,  CRDA1  becomes  the  desirable  choice  if  the  decision  is  based  upon  objective 
values  only  (i.e.,  a  =  1). 


Figure  2.  Sensitivity  Analysis  on  Example  3-1  CRDAs 

Example  3-1  demonstrates  that  CRDAs  can  be  evaluated  in  comparison  to  one  another 
in  a  systematic  fashion  to  determine  which  is  more  desirable  according  to  costs  and 
benefits.  Though  systematic  and  somewhat  rigorous,  the  analysis  “tool”  allows  for  expert 
input  (via  subjective  attribute  weightings,  subjective  evaluation  ratings,  and  the  relative 
importance  weighting  for  OMk)  as  shown  by  Step  4.  Thus,  a  balance  between  logic 
(tangible,  objective  measures)  and  “feel”  (intangibles,  subjective  benefits)  is  struck  when 
utilizing  the  multiattribute  decision  analysis  methodology. 
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Expectations  of  the  Model 


Now  that  the  methodology  has  been  demonstrated,  how  is  it  going  to  be  utilized  to 
accomplish  the  purposes  of  this  research  effort?  As  shown  above,  the  methodology 
provides  a  foundation  on  which  to  analyze  both  objective  and  subjective  CRDA  benefits. 
The  methodology  further  shows  that  a  comparison  of  alternatives  is  available  based  upon 
those  benefits.  Finally,  the  example  illustrates  that  the  model  provides  the  capability  to 
perform  sensitivity  analysis  by  varying  certain  inputs. 

The  same  principles  demonstrated  in  Example  3-1  will  be  applied  to  this  research 
effort’s  focus:  to  provide  AFRL  decision-makers  a  “tool”  for  evaluating  which  CRDA 
opportunities  should  be  pursued.  It  has  been  documented  in  the  previous  chapters  that 
WL  decision-makers  have  no  systematic  way  of  evaluating  CRD  As  versus  each  other. 
Introduction  of  this  methodology  fills  this  gap. 

In  order  to  satisfy  the  demonstrated  need,  several  pieces  of  data  are  incorporated  into 
the  model.  The  first  piece  of  data  is  the  objective  and  subjective  benefit  measurements  of 
65  WL  CRD  As.  This  portion  was  previously  collected  by  Captain  Mark  Davis 
(Appendices  A  and  C).  Second,  inputs  concerning  the  relative  importance  of  several 
factors  (a  and  the  relative  importance  of  subjective  benefits  to  each  other)  from  four 
AFRL  technology  transfer  focal  points  (formerly  in  WL)  are  included.  This  information 
was  obtained  from  the  focal  points  when  they  responded  to  the  questionnaire  located  in 
Appendix  B.  Upon  receipt  of  the  four  responses,  the  average  value  of  each  input  was 
determined.  It  is  these  average  values  that  will  be  used  in  Chapter  IV’s  analysis. 


58 


Finally,  the  65  CRD  As  are  categorized  according  to  four  Wright  Laboratory 
directorates  (Materials,  Propulsion,  Avionics,  and  Flight  Dynamics).  Categorization  is  a 
very  important  step  because  this  is  where  the  mechanics  of  the  methodology  are  applicable 
to  the  Air  Force  technology  transfer  process.  Categorization  is  made  according  to  the 
two-letter  office  symbol  of  the  responsible  office  because  this  mirrors  the  current 
organizational  structure  within  the  AFRL  as  technology  transfer  focal  points  (those  who 
are  responsible  for  the  decision  of  which  CRD  As  to  pursue).  Second,  if  a  certain  category 
has  more  than  1 1  CRD  As  (Materials  has  3 1  CRD  As  and  Flight  Dynamics  has  16  CRD  As 
in  the  data  set),  these  CRD  As  are  split  into  subcategories  with  a  maximum  of  1 1  CRD  As 
in  each.  This  is  to  approximate  reality  in  that  focal  points  most  likely  will  have  less  than 
1 1  alternatives  to  choose  from  at  any  given  point  in  time.  The  restriction  to  1 1  CRD  As 
enables  the  results  given  by  the  methodology  to  be  more  robust  as  the  weighted  evaluation 
(which  sums  to  one  over  all  alternatives)  is  not  spread  as  thin  as  it  would  if  all  possible 
CRD  As  within  a  category  were  examined  at  once. 

After  categorization  is  complete,  the  four  steps  illustrated  by  Example  3-1  take  place  in 
each  category: 

STEP  1:  Determine  the  OMk  for  each  CRD  A. 

STEP  2:  Determine  the  subjective  measurement  (SMk)  for  each  CRDA. 

STEP  3:  Determine  the  weighted  evaluations  for  each  alternative  (WEk). 

STEP  4:  Perform  Sensitivity  Analysis. 

1 .  The  sensitivity  analysis  to  be  performed  in  Chapter  IV  consists  of  varying  a 
from  0  to  1 .  The  WEk’s  obtained  will  illustrate  how  the  order  of  CRDA 
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preference  changes  as  objective  measures  vary  from  being  completely 
unimportant  (a=0)  to  being  the  only  measurement  of  importance  (a=l). 

2.  A  graph  will  be  developed,  showing  which  CRDA  is  the  optimal  choice  as  a 
takes  on  different  values. 

The  results  obtained  by  performing  these  steps  will  indicate  to  decision-makers  which 
CRD  As,  based  on  a  given  level  of  a,  provide  the  best  return  on  investment  in  each 
category.  Using  this  information,  decision-makers  can  look  at  upcoming  CRDA 
opportunities  to  see  how  its  objective  and  subjective  benefits  line  up  with  the  CRDA 
identified  as  best  by  the  methodology  in  a  particular  category. 

In  summation,  the  primary  expectation  is  that  the  model  will  help  decision-makers  to 
evaluate  and  compare  potential  technology  transfer  opportunities.  The  model  is  expected 
to  provide  the  ability  to  evaluate  past  CRDA  performance.  Secondarily,  the  model  is 
expected  to  provide  individual  decision-makers  flexibility  to  tailor  the  model  according  to 
their  preferences  and  assessment  of  the  importance  of  individual  benefits.  The  bottom  line 
is  that  the  model  is  expected  to  satisfy  the  objectives  of  this  research  effort,  to  evaluate 
past  CRDA  performance  and  allow  for  the  performance  of  sensitivity  analysis. 
Limitations 

There  are  several  limitations  associated  with  the  multiattribute  decision  analysis 
methodology  and  its  application  to  Air  Force  technology  transfer.  First,  the  methodology 
is  solely  based  upon  user  input.  It  assumes  we  have  captured  all  possible  objective  and 
subjective  benefits.  Second,  the  methodology  requires  experts  to  delineate  between  items 
that  might  not  easily  be  done  (e.g.,  is  morale  two  times  more  important  than  image,  three 
times,  etc.).  Third,  while  Example  3-1  contains  separate  input  values  for  direct  costs  and 


indirect  costs,  the  available  Wright  Laboratory  data  is  in  average  values.  These  critical 
pieces  of  information  (due  to  the  potential  to  distinguish  between  CRD  As)  are  a  “wash” 
for  each  alternative  in  this  instance.  Therefore,  they  will  be  excluded  from  the  actual 
exercising  of  the  methodology  in  chapter  IV.  Though  these  limitations  exist,  they  can  be 
overcome.  For  example,  benefits  associated  with  technology  transfer  can  be  ascertained 
through  periodic  questioning  of  area  experts.  Nevertheless,  the  methodology  provides 
decision-makers  with  a  tool  in  which  the  relative  ranking  of  technology  transfer 
opportunities  is  determinable. 

Summary 

This  chapter  has  introduced  the  methodology  to  be  utilized  in  fulfilling  the  objectives  of 
this  research  effort.  Data  collection  efforts  to  obtain  the  needed  inputs  for  the 
methodology  are  discussed.  This  chapter  also  provides  an  example  scenario  illustrating 
the  mechanics  of  the  procedure.  Finally,  Chapter  III  points  out  how  the  methodology’s 
use  will  satisfy  the  objectives  of  this  research  effort.  Chapter  IV  will  exercise  the 
methodology  using  the  “real”  data  gathered  concerning  technology  transfer  at  Wright 
Laboratory.  Chapter  V  will  be  composed  of  conclusions  and  recommendations  from  the 
results  gathered  by  using  the  model. 
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IV.  Analysis  and  Results 


Introduction 

The  objectives  of  this  research  effort  are  to  develop  an  ROI  model  for  technology 
transfer,  exercise  the  model  using  previous  CRD  As,  and  demonstrate  the  model’s 
capability  to  perform  sensitivity  analysis.  Chapter  III  illustrated  these  three  objectives 
using  hypothetical  data.  This  chapter  will  satisfy  the  research  objectives  by  using  actual 
data.  This  chapter  begins  with  an  overview  of  how  data  was  collected  and  is  followed  by 
results  obtained  by  applying  the  model  developed  in  Chapter  III  to  the  data. 

Data  Overview 

As  discussed  briefly  in  Chapter  III,  the  data  gathered  for  this  research  is  from  three 
primary  sources.  The  first  portion  of  data  is  the  objective  and  subjective  benefit 
measurements  of  Wright  Laboratory  CRD  As.  In  his  1997  research,  Captain  Mark  Davis 
collected  data  concerning  69  separate  CRD  As  involving  four  of  Wright  Laboratory’s 
seven  directorates.  The  objective  and  subjective  benefits  considered  are  those  listed  in 
Chapter  III  and  used  in  Example  3-1  (Appendix  A  contains  this  information). 

The  second  portion  of  data  gathered  concerns  the  relative  importance  of  the 
information  found  in  Davis’  work.  Several  factors  (relative  importance  of  objective 
benefits  [a]  and  the  relative  importance  of  subjective  benefits  to  each  other)  were  of  prime 
concern.  In  order  to  determine  the  relative  importance  of  the  aforementioned  variables, 
the  expert  opinion  of  four  AFRL  technology  transfer  focal  points  (formerly  in  WL)  was 
obtained.  Technology  transfer  focal  points  were  chosen  for  the  following  reasons. 

According  to  the  Air  Force  Program  Manager  for  the  Air  Force  Technology  Transfer 
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program,  focal  points  are  the  personnel  in  position  to  make  decisions  concerning  the 
pursuit  of  specific  CRD  As  (Guilfoos,  1998A).  More  specifically,  among  their  many 
duties,  transfer  focal  points  are  required  to  (Guilfoos,  1998C): 

A.  Identify  technologies  suitable  for  transfer. 

B.  Collect,  maintain,  and  report  all  data  elements  required  for  management  of 
transfer. 

C.  Prepare  technology  assessments. 

D.  Take  part  in  public  and  private  sector  opportunities  for  transfer. 

E.  Provide  and  disseminate  information  of  federally  owned  or  originated  products, 
processes,  services,  and  facilities. 

From  this  list  of  responsibilities  it  is  evident  that  transfer  focal  points  are  required  to  be  in 
the  middle  of  the  transfer  process  and  thus  are  in  a  position  to  determine  which 
opportunities  should  be  pursued. 

The  instrument  used  to  gather  focal  point  inputs  is  the  questionnaire  found  in 
Appendix  B.  Upon  receipt  of  the  four  responses,  the  average  value  for  each  input  (a  and 
relative  importance  of  subjective  benefits)  was  determined.  The  average  value  of  a 
(relative  importance  of  objective  benefits  to  subjective  benefits)  for  the  four  focal  points 
was  0.5625.  The  relative  importance  of  the  individual  subjective  benefits  to  each  other 
according  to  the  focal  points,  which  will  be  used  in  this  chapter’s  analysis  are  shown  in 
Table  8: 
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Table  8.  Relative  Importance  of  Subjective  Benefits 


SUBJECTIVE  BENEFIT 

RELATIVE  IMPORTANCE 

Productivity 

0.245  ] 

Resource  Utilization 

0.215 

Management 

0.125 

Image 

0.145 

Morale 

0.0825 

Technical  Capacity 

0.1875 

TOTAL 

1.0000 

Interestingly,  there  was  some  difference  in  opinion  between  the  focal  points  as  to  which 
subjective  benefits  were  most  important.  Table  9  provides  the  low,  high,  and  average 
values  given  for  each  subjective  benefit  category. 


Table  9.  Subjective  Benefit  Responses:  Low,  High,  Average 


BENEFIT 

LOW 

HIGH 

AVERAGE 

0.16 

0.32 

0.245 

Resource  Util. 

0.15 

0.30 

0.215 

0.05 

0.20 

0.125 

0.10 

0.25 

0.145 

Morale 

0.05 

0.16 

0.0825 

Technical  Cap. 

0.05 

0.25 

0.1875 

TOTAL 

1.0000 

The  chart  indicates  the  biggest  discrepancies  occurred  in  the  importance  of  technical 
capability  (a  difference  of  0.20  from  low  to  high  response)  and  the  smallest  difference 
occurred  with  morale  (0. 1 1  from  low  to  high  response).  The  remaining  four  benefits  had 
differences  of  either  0.15  of  0.16  from  the  low  to  high  response. 

The  third  and  final  source  of  data  is  the  categorization  of  the  CRD  As  from  the  first 
step.  The  CRD  As  identified  were  sorted  into  four  separate  categorizes  representing  four 
Wright  Laboratory  directorates  (Materials,  Flight  Dynamics,  Propulsion,  and  Avionics). 
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Four  CRDAs  did  not  fit  into  the  primary  categories  and  were  deleted.  The  65  CRDAs 
were  segregated  into  categories  according  to  the  “two-letter”  office  symbol  of  the 
responsible  office.  This  approach  resulted  in  the  distribution  of  CRDAs  shown  in 
Table  10: 


Table  10.  CRDAs  per  Directorate 


DIRECTORATE 

NUMBER  OF  CRDAs 

Materials 

31 

Flight  Dynamics 

16 

Propulsion 

9 

Avionics 

9 

TOTAL 

65 

The  Materials  category  is  arbitrarily  broken  into  three  separate  sections  of  1 1,  10,  and 
10  CRDAs,  respectively.  Similarly,  Flight  Dynamics  is  broken  into  two  separate  groups  of 
8  CRDAs  each.  There  are  several  reasons  for  this  further  breakdown  within  category. 

First,  limiting  the  number  of  CRDAs  being  compared  to  one  another  prevents  the  results 
obtained  from  being  diluted.  In  other  words,  since  rankings  are  based  according  to 
weighted  values,  splitting  that  weight  over  less  CRDAs  provides  better  results.  Second, 
smaller  groups  of  CRDAs  are  more  representative  of  reality.  A  focal  point  faces  the 
decision  to  pursue  transfer  opportunities  at  a  point  in  time,  not  over  years.  Because  of 
this,  it  is  realistic  to  have  only  a  handful  of  potential  alternatives  to  pursue  (not  16,  and 
definitely  not  31). 

It  is  realized  that  there  are  many  other  methods  available  to  splitting  the  categories 
(chronological,  three-letter  office  symbol,  etc.).  However,  in  an  effort  to  protect  the 
confidentiality  of  those  individuals  providing  information,  delineation  of  the  data  points  is 
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stopped  at  the  “two-letter”  (or  directorate)  level.  If  the  data  points  were  further  broken 
down  to  the  “three-letter”  level,  it  would  be  possible  to  track  the  CRDA  to  the  specific 
individual  within  the  directorate  who  was  responsible  for  the  transfer.  This  approach  is 
inappropriate  due  to  the  restrictions  associated  with  the  data  garnered  and,  therefore, 
further  category  breakdown  was  accomplished  by  randomly  assigning  CRD  As  to 
individual  subcategories. 

Data  Analysis 

Now  that  the  origins  of  the  data  have  been  described  as  well  as  the  determination  of 
categories,  the  model  is  exercised  on  the  seven  categories:  Materials  (3),  Flight  Dynamics 
(2),  Propulsion  (1),  Avionics  (1). 

Materials  Directorate 

The  Materials  Directorate  “explores  new  materials  and  processes  for  advanced 
aerospace  applications.  Its  current  focus  is  on  thermal  protection  materials,  metallic  and 
nonmetallic  structural  materials,  aerospace  propulsion  materials,  electromagnetic  and 
electronic  materials  and  laser-hardened  materials”  (Wright-Patterson  Air  Force  Base, 

1 995 : 1 3).  One  of  the  most  familiar  technologies  developed  in  this  directorate  is 
composite  materials  that  are  used  on  military  and  civilian  aircraft. 

As  stated  earlier,  there  are  3 1  Materials  CRD  As  in  the  data  set.  The  CRD  As  are 
arbitrarily  split  into  three  separate  groups:  A,  B,  and  C.  The  four  steps  of  the  Canada  and 
Sullivan  methodology  are  systematically  applied  to  each  group  individually.  This 
procedure  is  conducted  on  each  subsequent  category  as  well. 
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Materials  A 


Steps  1-3:  Obtain  objective  and  subjective  measurements,  weighted  evaluations. 


Table  1 1  indicates  the  results  of  step  three  for  the  Materials  A  category  using  the  a  of 


0.5625  obtained  from  the  technology  transfer  focal  points: 


Table  11.  Materials  A  Objective,  Subjective,  and  Weighted  Evaluations 


MATERIALS  A 

OM„  = 

SMk  = 

II 

je 

i 

CRDA1 

0.0000 

0.0607 

CRDA2 

0.0000 

0.0071 

0.0031 

CRDA3 

0.0000 

0.0106 

0.0046 

CRDA4 

0.5048 

0.2346 

0.3865 

CRDA5 

0.0190 

0.0825 

0.0468 

CRDA6 

0.0000 

0.0494 

0.0216 

CRDA7 

0.0000 

0.0494 

0.0216 

CRDA8 

0.0000 

0.0494 

0.0216 

CRDA9 

0.0381 

0.1834 

0.1017 

CRDA  10 

0.0000 

0.0106 

0.0046 

CRDA11 

0.4381 

0.2623 

0.3612 

TOTAL 

1.0000 

1.0000 

1.0000 

On  the  basis  of  using  the  focal  points’ 


value  for  a,  CRDA4  appears  to  be  the  optimal 


choice  of  the  1 1  CRD  As  within  the  Materials  A  category  (WEk  =  0.3865),  though 


CRDA1 1  is  nearly  as  good  with  a  weighted  evaluation  value  of  0.3612.  How  does  this 


relationship  change  as  a  is  allowed  to  take  on  values  ranging  from  0  to  1?  In  other  words, 


will  CRDA4  remain  the  alternative  of  choice  as  the  importance  of  objective  measurements 


are  allowed  to  range  from  completely  unimportant  to  solely  important?  Step  four  of  the 
methodology  reveals  this  answer.  (Note:  The  total  column  values  of  one  simply  indicate 
that  the  totals  for  objective  measurements,  subjective  measurements,  and  weighted 
evaluation  values  are  one.  This  practice,  a  “sanity  check”,  is  continued  throughout  the 


remaining  six  cases). 
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STEP  4:  Perform  Sensitivity  Analysis. 

Sensitivity  analysis  is  performed  by  allowing  a  to  vary  from  0  (objective  benefits  being 
completely  unimportant)  to  1  (objective  benefits  being  completely  important).  (Note: 

This  practice  is  also  followed  in  the  remaining  six  cases).  Figure  3  illustrates  how  the 
optimal  CRDA  choice  changes  as  objective  benefits  range  from  being  unimportant  to  all- 
important. 
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Figure  3.  Sensitivity  Analysis  on  Materials  A  CRDAs 
Figure  3  indicates  that  CRDA4  is  the  superior  choice  as  long  as  a  >  0.2  (in  other  words, 
subjective  benefits  are  not  greater  than  80%  of  the  overall  importance).  However,  if  the 
importance  of  objective  benefits  falls  below  the  20%  level,  CRDA1 1  becomes  the  optimal 
choice.  Of  interest  also  is  the  fact  that  the  remaining  nine  CRDAs  are  not  even  close  to 
being  the  optimal  choices,  no  matter  the  value  of  a.  Therefore,  depending  upon  the 
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importance  of  objective  benefits,  the  optimal  choice  is  either  CRDA4  or  CRDA1 1 .  If  the 
budget  enables  the  pursuit  of  three  CRD  As,  the  choice  of  CRDA  to  accompany  CRD  As  4 
and  1 1,  at  any  level  of  a,  is  CRDA9. 

Materials  B 

Steps  1-3:  Obtain  objective  and  subjective  measurements,  weighted  evaluations. 

The  second  category  of  material  results  for  steps  one  through  three  are  shown  in 
Table  12: 


Table  12.  Materials  B  Objective,  Subjective,  and  Weighted  Evaluations 


MATERIALS  B 

OMk  = 

SMk  = 

WEk  = 

CRDA  12 

0.1670 

0.2200 

0.1902 

CRDA  13 

0.2783 

0.2151 

0.2507 

CRDA  14 

0.1237 

0.1183 

0.1214 

CRDA  15 

0.1237 

0.1187 

0.1215 

CRDA  16 

0.1328 

0.1370 

0.1347 

CRDA  17 

0.0000 

0.0000 

0.0000 

CRDA  18 

0.0508 

0.0253 

0.0396 

CRDA  19 

0.0000 

0.0000 

0.0000 

CRDA  20 

0.0618 

0.0756 

0.0679 

CRDA  21 

0.0618 

0.0900 

0.0742 

TOTAL 

1.0000 

1.0000 

1.0000 

The  results  in  Table  12  indicate  that  CRDA13  has  the  highest  weighted  evaluation  overall 


(0.2507)  with  CRDA12  following  (0. 1902).  There  is  a  group  of  CRD  As  (14,  15,  and  16) 


that  are  all  at  the  next  level.  The  remaining  CRD  As  appear  to  be  unattractive  options. 


STEP  4:  Perform  Sensitivity  Analysis. 

Figure  4  shows  how  the  order  of  CRDA  preference  changes  as  a  varies  from  0  to  1 . 
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Figure  4.  Sensitivity  Analysis  on  Materials  B  CRD  As 

It  is  apparent  from  Figure  4  that  CRDA13  is  superior  at  every  point  where  a  >  0.05.  In 
other  words,  as  long  as  subjective  benefits  are  not  more  than  95%  of  overall  benefit 
importance,  CRDA13  is  the  optimal  choice  according  to  the  model.  Additionally, 

CRDA13  becomes  more  dominant  is  a  increases.  Figure  4  also  reveals  that  CRDA12  is 
consistently  second  choice  indicating  a  robust  choice.  It  is  also  shown  that  CRDA16  is 
consistently  third.  While  CRD  As  13  and  12  have  some  vertical  slope  to  them,  CRDA  16’s 
weighted  evaluation  line  appears  nearly  horizontal  (i.e.,  slope  equal  to  zero).  The  reason 
for  this  is  that  CRDA16  is  solid  in  both  the  objective  and  subjective  measures.  This  means 
the  CRDA  not  only  provides  monetary  rewards,  but  subjective  benefits,  too.  CRDAs  14 
and  15  are  the  next  choice  of  alternatives.  They  are  nearly  identical  to  each  other  and  both 
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have  nearly  horizontal  slopes,  similar  to  CRDA16.  After  these  two  CRD  As,  the  remaining 
options  appear  to  be  nondesirable. 

Materials  C 

Steps  1-3;  Obtain  objective  and  subjective  measurements,  weighted  evaluations. 

Table  13  indicates  the  results  of  step  three  for  the  Materials  C  category  using  an  a  of 
0.5625: 


Table  13.  Materials  C  Objective,  Subjective,  and  Weighted  Evaluations 


MATERIALS  C 

OMk  = 

SMk  = 

WEk  = 

CRDA  22 

0.0000 

0.0782 

0.0342 

CRDA  23 

0.0000 

0.0078 

0.0034 

CRDA  24 

0.0000 

0.0118 

0.0051 

CRDA  25 

0.8983 

0.3974 

0.6792 

CRDA  26 

0.0339 

0.1003 

0.0630 

CRDA  27 

0.0000 

0.0534 

0.0234 

CRDA  28 

0.0000 

0.0534 

0.0234 

CRDA  29 

0.0000 

0.0534 

0.0234 

CRDA  30 

0.0678 

0.2323 

0.1398 

CRDA 31 

0.0000 

0.0118 

0.0051 

TOTAL 

1.0000 

1.0000 

1.0000 

Table  13  indicates  CRDA25  (WEk  =  0.6792)  is  by  far  the  most  desirable  of  the  10  CRD  As 


within  this  category.  The  only  other  CRDA  with  a  weighted  evaluation  value  of  any 


significance  is  CRDA30  (WEk  -  0.1398). 


STEP  4;  Perform  Sensitivity  Analysis. 


Figure  5  is  the  basis  for  sensitivity  analysis  in  category  Materials  C. 
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Figure  5.  Sensitivity  Analysis  on  Materials  C  CRDAs 

Figure  5  indicates  that  one  alternative  is  far  and  away  the  most  desirable  choice.  That 
alternative  is  CRDA25,  the  optimal  choice  at  all  levels  of  a.  Interestingly,  CRDA25  is  the 
only  choice  that’s  desirability  increases  with  a.  This  means  that  while  CRDA25  is 
superior  to  all  other  alternatives  on  the  basis  of  subjective  benefits  alone,  CRDA25  is  even 
more  superior  to  the  remaining  choices  when  objective  benefits  are  considered  alone. 

After  the  optimal  choice,  CRD  A3  0  is  the  consistently  second.  However,  its  position  as 
second  decreases  in  strength  as  a  increases.  Finally,  the  remaining  CRDAs  appear  to  be 
equally  undesirable. 
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Flight  Dynamics  Directorate 

The  Flight  Dynamics  Directorate  “conducts  a  foil  spectrum  of  flight  vehicle  research. 
Primary  areas  of  interest  include  aircraft  structures,  vehicles  subsystems  (e.g.,  landing 
gear,  transparencies),  flight  control,  and  aeromechanics”  (Wright-Patterson  Air  Force 
Base,  1995: 12).  Additionally,  Flight  Dynamics  is  the  directorate  responsible  for 
experimental  test  vehicles  used  to  demonstrate  integrated  technologies  in  the  air  (Wright- 
Patterson  Air  Force  Base,  1995:12). 

The  Flight  Dynamics  directorate  has  16  CRD  As  in  the  data  set.  The  CRD  As  are 
arbitrarily  split  into  two  groups  of  eight.  The  four  steps  to  the  methodology  are 
systematically  applied  to  the  Flight  Dynamics  directorate  CRD  As. 

Flight  Dynamics  A 

Steps  1-3:  Obtain  objective  and  subjective  measurements,  weighted  evaluations. 

Table  14  indicates  the  results  of  step  three  for  the  Flight  Dynamics  A  category: 


Table  14.  Flight  Dynamics  A  Objective,  Subjective,  and  Weighted  Evaluations 


Flight  Dynamics  A 

OMk  = 

SMk  = 

WEk  = 

CRDA  1 

0.2994 

0.1798 

0.2471 

CRDA2 

0.0000 

0.2174 

0.0951 

CRDA  3 

0.6956 

0.3765 

0.5560 

CRDA  4 

0.0000 

0.0000 

CRDA  5 

0.0000 

0.0000 

0.0000 

CRDA  6 

0.0000 

0.0673 

0.0294 

CRDA  7 

0.0000 

0.0000 

0.0000 

CRDA  8 

0.0050 

0.1590 

0.0724 

TOTAL 

1.0000 

1.0000 

1.0000 
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CRD  A3,  by  far,  is  the  optimal  choice  in  this  group  (WEk  =  0.5560).  The  second  choice  is 
CRDA1  (WEk  =  0.2471).  It  does  not  appear  this  relationship  would  vary  as  a  changed 
from  0  to  1.  However,  step  four  indicates  whether  or  not  this  is  correct. 

STEP  4:  Perform  Sensitivity  Analysis. 

Sensitivity  Analysis  for  Flight  Dynamics  A  is  performed  with  the  help  of  Figure  6. 
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Figure  6.  Sensitivity  Analysis  on  Flight  Dynamics  A  CRDAs 
The  results  of  Figure  6  indicate  that  CRD  A3  is  always  the  optimal  choice.  The  figure  also 
indicates  that  CRD  A3  becomes  even  more  dominant  as  objective  measures  become  more 
important.  The  second  choice  (in  all  cases  where  a  >  0)  is  CRDA1 .  CRDA1  follows  the 
same  relationship  as  CRD  A3  as  it  too  gets  stronger  as  objective  measures  increase  in 
importance.  The  remaining  CRDAs  follow  a  pattern  opposite  to  CRDAs  3  and  1.  Their 
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weighted  evaluation  values  become  smaller  as  objective  measures  become  more  important. 
It  is  noted  though  that  CRDA2  finishes  third  at  all  levels  of  a  >  0.05. 

Flight  Dynamics  B 

Steps  1-3:  Obtain  objective  and  subjective  measurements,  weighted  evaluations. 

Table  15  indicates  the  results  of  steps  one  through  three  for  the  Flight  Dynamics  B 
category  using  an  a  of  0.5625: 


Table  15.  Flight  Dynamics  B  Objective,  Subjective,  and  Weighted  Evaluations 


Flight  Dynamics 

B 

OMk  = 

SMk  = 

WEk  = 

CRDA9 

0.0000 

0.0747 

0.0327 

CRD  A  10 

0.0000 

0.0243 

0.0106 

CRDA11 

0.7612 

0.1212 

0.4812 

CRDA  12 

0.0000 

0.0000 

0.0000 

CRD  A  13 

0.0000 

0.2100 

0.0919 

CRDA  14 

0.0896 

0.1285 

0.1066 

CRDA  15 

0.0000 

0.0855 

0.0374 

CRDA  16 

0.1493 

0.3558 

0.2396 

TOTAL 

1.0000 

1.0000 

1.0000 

CRDA1 1  is  the  superior  choice  in  this  group  with  a  weighted  evaluation  value  of  0.4812. 


The  second  best  choice  is  CRDA16,  which  has  a  weighted  evaluation  value  over  double 


the  next  best  CRD  As,  13  and  14. 


STEP  4:  Perform  Sensitivity  Analysis 


Figure  7  contains  the  results  of  performing  sensitivity  analysis  on  Flight  Dynamics  B 


CRD  As. 
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Figure  7.  Sensitivity  Analysis  on  Flight  Dynamics  B  CRDAs 

Figure  7  indicates  results  that  are  more  complicated  than  those  illustrated  in  the  previous 
four  categories.  There  are  many  more  places  where  alternatives  cross  each  other.  These 
points  of  intersection  indicate  where  one  CRD  A  becomes  more  optimal  than  the  other. 
According  to  the  figure,  CRDA1 1  is  fourth  most  desirable  when  a  is  0.  However,  when  a 
increases  to  0.3,  CRDA1 1  vaults  into  first  place.  CRDA1 1  is  the  only  one  that  increases 
as  objective  measurements  gain  more  importance  relative  to  subjective  benefits.  CRDA16 
is  a  solid  choice,  always  ranking  either  first  or  second.  The  CRDAs  sitting  in  third  and 
fourth  place  depends  upon  the  value  of  a.  CRDAs  13  and  14  have  relatively  the  same 
weighted  evaluations,  but  14  is  more  stable  across  the  gamut  of  alpha  (less  slope)  whereas 
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13 ’s  strength  is  when  a  is  small.  Therefore,  the  choice  of  third  most  optimal  alternative 
depends  upon  the  relative  importance  of  objective  and  subjective  measures. 

Propulsion  Directorate 

The  Propulsion  Directorate  (also  known  as  the  Aero  Propulsion  and  Power 
Directorate)  “focuses  on  air-breathing  propulsion  and  aerospace  power  technology,  which 
includes  fuels  and  lubricants,  turbine  engines,  and  high-performance,  high-Mach,  air- 
breathing  propulsion  applications.  Aerospace  power  research  centers  around 
electrochemical  energy  storage,  hyperconducting  generators,  and  power  conditioning 
subsystems”  (Wright-Patterson  Air  Force  Base,  1995:12).  The  Propulsion  directorate  has 
9  CRD  As  associated  with  this  research.  The  multiattribute  decision  analysis  methodology 
will  now  be  applied  to  the  propulsion  CRD  As. 

Steps  1-3:  Obtain  objective  and  subjective  measurements,  weighted  evaluations. 

Table  16  indicates  the  results  for  the  propulsion  category  using  an  a  of  0.5625: 


Table  16.  Propulsion  Objective,  Subjective,  and  Weighted  Evaluations 


PROPULSION 

OMk  = 

SMk  = 

WEk  = 

CRDA1 

0.0094 

0.1672 

CRDA2 

0.0000 

0.0000 

0.0000 

CRDA3 

0.0000 

0.0809 

0.0354 

CRDA4 

0.0003 

0.1078 

0.0473 

CRDA5 

0.9870 

0.0415 

0.5733 

CRDA6 

0.0000 

0.3766 

0.1647 

CRDA7 

0.0000 

0.1068 

0.0467 

CRD  A  8 

0.0002 

0.0280 

0.0124 

CRDA9 

0.0031 

0.0912 

0.0416 

TOTAL 

1.0000 

1.0000 

1.0000 
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CRDA5  is  the  superior  choice  in  this  group  with  a  weighted  evaluation  value  of  0.5733. 
CRDA6  is  the  only  other  option  with  a  considerable  weighted  evaluation  value  (0.1647). 
STEP  4:  Perform  Sensitivity  Analysis 

Figure  8  contains  the  basis  for  sensitivity  analysis  of  Propulsion  CRD  As. 
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Figure  8.  Sensitivity  Analysis  on  Propulsion  CRDAs 

Figure  8  presents  results  that  are  similar  to  those  in  the  previous  category  as  illustrated  by 
Figure  7.  The  main  point  of  comparison  is  the  importance  of  the  value  of  a.  Both 
categories  contain  a  CRDA  with  a  steep  positive  slope  associated  with  the  weighted 
evaluation  line.  In  the  case  of  Figure  7,  CRDA1 1  possesses  the  steep  positive  slope.  In 
this  category,  CRDA5  has  the  steep  positive  slope.  In  fact,  CRDA5  is  the  only  choice 
with  a  positive  slope.  Consequently,  as  objective  measures  become  of  greater  importance, 
CRDA5  is  optimal  choice  by  a  greater  distance.  Figure  8  indicates  CRDA5  is  not  the 
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superior  alternative  when  a  is  low;  however,  once  a  rises  past  0.15,  CRDA5  maintains  the 
top  ranking.  When  a  is  low,  CRDA6  is  the  superior  alternative.  It  should  be  noted  that 
CRDA6  is  always  second  best,  representative  of  a  solid,  robust  option.  CRDA1  is  second 
best  with  subjective  measures  only  (a=0). 

Avionics  Directorate 

The  Avionics  Directorate  is  the  final  one  analyzed  in  this  effort.  The  Avionics 
Directorate  “conducts  research  and  development  activities  in  the  fields  of  offensive 
sensors  (e.g.,  radar,  infrared  search  and  track,  forward  looking  infrared),  weapon  delivery 
systems,  reconnaissance,  electronic  warfare,  navigation,  communications,  and  avionics 
integration”  (Wright-Patterson  Air  Force  Base,  1995: 13).  There  are  nine  avionics  CRD  As 
in  this  study.  The  Canada/Sullivan  methodology  is  applied  to  them  next. 

Steps  1-3;  Obtain  objective  and  subjective  measurements,  weighted  evaluations. 

Table  17  indicates  the  results  of  step  three  for  the  Avionics  category: 


Table  17.  Avionics  Objective,  Subjective,  and  Weighted  Evaluations 


AVIONICS 

OMk  = 

SM„  = 

WE„  = 

CRDA1 

0.2119 

0.0622 

0.1464 

CRDA2 

0.1271 

0.2097 

0.1632 

CRD  A  3 

0.0000 

0.1523 

0.0666 

CRDA4 

0.0424 

0.1829 

0.1038 

CRDA5 

0.0254 

0.3657 

0.1743 

CRDA6 

0.0000 

0.0000 

0.0000 

CRDA7 

0.0424 

0.0000 

0.0238 

CRDA8 

0.1059 

0.0000 

0.0596 

CRDA9 

0.4449 

0.0272 

0.2622 

TOTAL 

1.0000 

1.0000 

1.0000 
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CRDA9  is  the  optimal  choice  in  the  avionics  category  with  a  weighted  evaluation  value  of 
0.2622.  However,  there  are  three  CRD  As  that  are  relatively  close  in  the  race  for  second 
place  (CRDA5,  CRDA2,  and  CRDA1). 

STEP  4:  Perform  Sensitivity  Analysis. 

Figure  9  indicates  how  the  avionics  CRD  As  weighted  evaluation  values  change  as  a 
ranges  from  0  to  1. 
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Figure  9.  Sensitivity  Analysis  on  Avionics  CRD  As 

Situations  such  as  the  one  illustrated  in  Figure  9  highlight  the  flexibility  of  the 
multiattribute  decision  analysis  methodology.  It  highlights  the  flexibility  of  the  tool  in  that 
the  order  of  preference  truly  depends  on  the  a  chosen  by  the  decision-maker.  For 
example,  if  a  =  0.8,  then  the  top  5  in  order  of  decreasing  preference  is:  9,  1,  2,  5,  and  8. 
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However,  if  a=  0.2,  then  order  of  the  top  5  changes  to:  5,  2, 4,  3,  and  9.  By  simply 
comparing  the  numbers,  one  can  see  that  this  is  a  big  difference.  CRDA1,  second  best  at 
a=0.8  is  no  longer  in  the  top  5  at  the  new  a.  The  same  holds  true  for  CRDA8  which  does 
not  appear  in  the  top  5,  being  replaced  by  CRDA4. 

In  contrast  to  the  previous  six  categories,  Avionics  CRD  As  all  possess  relatively  steep 
slopes.  These  slopes  indicate  a  difference  in  the  CRD  As  ranking  versus  each  other  as 
objective  measurements  become  more  important  relative  to  subjective  measurements  (as  a 
increases).  This  indicates  that  each  particular  CRDA  is  not  as  robust  across  all  levels  of  a 
as  CRD  As  with  slopes  near  zero  (a  horizontal  line). 

Summary 

This  chapter  applied  the  multiattribute  decision  analysis  methodology  to  65  Wright 
Laboratory  CRD  As  with  appropriate  comments  made  for  each  category.  The 
methodology  is  shown  to  be  applicable  to  this  arena  as  well  as  capable  of  providing  the 
manager  flexibility  in  decision-making. 

The  results  obtained  through  the  use  of  the  multiattribute  decision  analysis 
methodology  varied  in  level  of  complexity.  The  first  directorate  (Materials)  contained  31 
CRD  As  that  were  split  into  three  separate  groups.  The  results  of  all  three  groups 
indicated  that  the  CRD  As  retained  their  rankings  in  relation  to  each  other  over  the  range 
of  a.  This  implies  that  choosing  CRD  As  in  these  groupings  was  relatively  simple.  The 
second  directorate  (Flight  Dynamics)  had  16  CRD  As  which  were  split  into  two  groups  of 
eight.  In  contrast  to  the  Materials  CRD  As,  Flight  Dynamics  CRD  As  possessed  less  “cut- 
and-dried”  results.  Specifically,  the  Flight  Dynamics  B  category  contained  CRDA1 1 
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which,  depending  on  the  level  of  a,  ranged  from  being  the  fourth  most  desirable  option  to 
the  first.  The  third  directorate  (Propulsion)  contained  nine  CRDAs.  Propulsion’s  results 
were  similar  to  that  of  Flight  Dynamics  B.  CRDA5  of  this  category  ranged  from  third 
place  to  most  optimal  as  a  took  upon  different  values.  The  final  category  (Avionics) 
provided  results  on  the  opposite  end  of  the  spectrum  from  those  observed  in  the  first 
category  (Materials).  Analysis  was  performed  on  nine  Avionics  CRDAs.  The  results 
cannot  be  summed  up  as  quickly  here  as  they  were  for  the  other  three  directorates.  As 
pictured  in  Figure  9,  there  are  many  points  at  which  the  weighted  evaluation  lines  for 
CRDAs  cross  each  other.  As  stated  earlier,  it  is  at  these  points  of  intersection  where  the 
order  of  preference  between  CRDAs  changes.  Figure  9  highlights  the  importance  of  this 
methodology  to  decision-makers.  The  order  of  CRDA  preference  depends  on  the  relative 
importance  the  decision-maker  places  upon  objective  measures.  It  is  this  order  that  the 
methodology  reveals. 

Looking  forward,  Chapter  V  will  contain  conclusions  concerning  the  methodology 
including  identified  strengths,  limitations,  and  usage.  Additionally,  managerial  implications 
will  be  drawn  from  the  results  of  Chapter  IV.  Recommendations  of  future  research  topics 
as  well  as  an  overall  summary  will  conclude  this  research  effort. 
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V.  Conclusion 


Introduction 

According  to  the  Air  Force  Materiel  Command  Technology  Transfer  Handbook,  the 
purpose  for  participation  in  technology  transfer  is  to  maximize  return  on  investment  in 
RDT&E  (1995:A-5).  However,  the  Air  Force  currently  has  no  model  in  place  to  measure 
the  return  on  investment  of  transfer  opportunities  (Guilfoos,  1998 A).  This  research  was 
performed  in  an  effort  to  eliminate  this  gap.  To  this  extent,  the  purpose  of  this  research 
effort  as  outlined  at  the  outset  in  Chapter  I  was  to  satisfy  the  following  three  research 
objectives: 

A.  Propose  a  ROI  model  to  be  used  by  Air  Force  decision  makers  in  performing 
Technology  Transfer; 

B.  Evaluate  the  Return  on  Investment  (ROI)  of  past  Technology  Transfers; 

C.  Perform  Sensitivity  Analysis  on  Technology  Transfers  when  some  inputs  are 
uncertain. 

The  first  objective  was  satisfied  with  the  introduction  of  the  multiattribute  decision 
analysis  methodology  in  Chapter  II  and  a  hypothetical  demonstration  of  its  application  in 
Chapter  HI.  The  second  objective  was  satisfied  in  Chapter  IV  as  the  methodology  was 
applied  to  65  Wright  Laboratory  Cooperative  Research  &  Development  Agreements 
(CRDAs).  The  third  objective  was  satisfied  through  the  analysis  of  the  CRD  As. 
Sensitivity  analysis  was  performed  by  varying  the  parameter  that  measured  the  relative 
importance  of  objective  benefits  versus  subjective  benefits  (a).  Therefore,  by  satisfying 
the  three  research  objectives,  this  effort  provides  Air  Force  decision-makers  a  model  on 
which  to  determine  the  ROI  associated  with  each  CRD  A. 
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In  the  process  of  satisfying  the  primary  research  objectives,  several  interesting 
conclusions  resulted  which  will  be  discussed  next.  They  are  followed  by  the  managerial 
implications  of  those  results.  A  discussion  of  areas  for  future  research  as  well  as  an 
overall  summary  will  conclude  this  effort. 

Conclusions 

Given  that  this  methodology  is  the  first  attempt  to  simultaneously  measure  the 
quantitative  and  qualitative  benefits  of  Air  Force  technology  transfer,  there  are  both 
strengths  and  limitations.  These  strengths  and  limitations  apply  not  only  to  the  model,  but 
also  to  the  gathering  of  inputs  for  the  model.  These  areas  are  discussed  below. 

Strengths 

1.  Applicability  &  ability  to  provide  information  that  promotes  decision-making  in 
a  variety  of  circumstances:  The  multiattribute  decision  analysis  methodology  presented 
by  Canada  and  Sullivan  is  applicable  to  the  Air  Force  for  usage  in  determining  the  ROI  of 
individual  technology  transfer  opportunities.  The  methodology  enables  leaders  to  make 
decisions  by  striking  a  balance  between  “hard  and  fast  numbers”  (objective  benefits)  and 
“gut  feel”  (subjective  benefits).  The  systematic  methodology  can  be  successfully  applied 
to  distinguish  between  alternatives  on  the  basis  of  the  objective  and  subjective  benefits 
each  option  possesses  in  comparison  to  one  another.  The  results  of  applying  the 
methodology  to  seven  separate  categories  in  Chapter  IV  show  the  methodology  is  able  to 
handle  a  long  continuum  of  scenarios.  On  one  extreme,  the  methodology  can  illuminate  a 
situation  where  there  is  a  definite  superior  option  no  matter  the  importance  placed  on 
objective  versus  subjective  benefits  (Material  C,  Flight  Dynamics  A  category).  On  the 
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other  extreme,  the  methodology  highlights  situations  where  the  choice  is  not  so  clear-cut 
and  depending  upon  how  the  decision-maker  values  objective  benefits  the  CRD  A  of 
preference  changes  (Flight  Dynamics  B,  Avionics). 

2.  Built  on  related  research  provided  bv  experts;  The  inputs  utilized  to  exercise  the 
model  in  Chapter  IV  were  obtained  from  Air  Force  technology  transfer  experts.  First,  the 
determination  of  the  objective  and  subjective  benefits  experienced  by  the  Air  Force  in 
technology  transfer  were  developed  in  the  research  by  Braun  and  Davis.  In  addition  to 
benefit  definition,  the  amount  of  benefits  associated  with  each  CRD  A  were  gathered  by 
Davis.  Objective  benefits  were  measured  in  dollars,  while  subjective  benefits  were 
measured  according  to  a  5-point  Likert  scale.  Additionally,  technology  transfer  focal 
points  (the  ultimate  users  of  this  model)  were  summoned  to  provide  their  estimation  of  the 
relative  importance  of  benefits  to  each  other. 

Limitations 

1 .  Proprietary  Data;  Several  limitations  are  associated  with  the  fact  that  all  data 

for  this  effort  was  collected  with  the  assurance  of  confidentiality.  Consequently,  this  led 
to  splitting  up  the  data  concerning  the  Materials  and  Flight  Dynamics  directorates  at  the 
“three  letter”  (directorate)  level  in  an  arbitrary  fashion.  Therefore,  the  results  obtained  in 
Chapter  IV  are  not  to  be  directly  applied,  but  instead  are  meant  to  show  that,  given  the 
necessary  data,  a  ranking  of  transfer  opportunities  can  be  obtained. 

2.  Validation:  Closely  related  to  the  first  weakness  is  the  idea  of  validation.  It  is  not 
feasible  to  validate  these  particular  results.  There  are  several  reasons  for  this  statement. 
First,  the  results  are  subject  to  splitting  the  CRD  As  from  directorates  into  smaller  groups 
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in  an  arbitrary  fashion.  Second,  the  CRD  As  were  from  directorates  in  Wright  Laboratory 
which  have  since  been  absorbed  piecemeal  into  the  Air  Force  Research  Laboratory. 

Though  there  are  limitations  related  to  the  specific  results  obtained  in  this  effort,  it  is 
undeniable  that  the  methodology  provides  a  foundation  from  which  decision-makers  can 
choose  one  alternative  over  another.  This  idea  is  further  delineated  in  the  managerial 
implications  section. 

Managerial  Implications 

In  his  1994  research,  West’s  primary  objective  was  to  determine  why  Wright 
Laboratory  participated  in  technology  transfer  (1994:54).  Among  the  conclusions  drawn, 
West  found  that  technology  transfer  success  was  hard  to  define,  technology  transfer  is 
difficult  to  measure  in  the  near  and  long  term,  and  that  no  one  metric  completely  measures 
technology  transfer  effectiveness.  These  shortcomings,  to  a  large  degree,  still  exist  in 
1998  (Guilfoos,  1998 A).  Braun  and  Davis  have  taken  strides  to  define  the  benefits 
associated  with  technology  transfer.  The  methodology  introduced  and  demonstrated  in 
this  research  effort,  in  large  part,  closes  the  loop  by  utilizing  their  findings. 

This  closure  is  especially  important  for  two  reasons:  (1)  once  again,  the  goal  of 
technology  transfer  participation  is  to  maximize  ROI  on  RDT&E  (AFMC  Handbook, 
1995.A-5);  and  (2)  the  Stevenson- Wydler  Technology  Innovation  Act  of  1980  requires 
that  a  minimum  of  0.5%  an  organization’s  R&D  budget  is  to  be  spent  on  activities  related 
to  technology  transfer  (AFMC  Handbook,  1995:A-2).  This  means  money  is  going  to  be 
spent  and  that  it  is  supposed  to  be  spent  in  the  way  that  provides  the  largest  possible 
return  on  investment. 
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The  bottom  line  is  this:  the  methodology  enables  decision-makers  to  use  both 
objective  and  subjective  benefit  measurements  to  determine  which  CRD  As  to  enter  into. 
They  now  have  a  way  to  determine  the  ROI  associated  with  transfer  opportunities  so  that 
they  can  make  educated  decisions  as  opposed  to  relying  on  instinct. 

Future  Research  Recommendations 

The  benefit  of  this  research  effort  is  that  it  provides  decision-makers  with  a  tool  to 
assess  ROI,  not  specific  results  on  which  to  make  future  transfer  choices.  There  are 
several  reasons  this  comment  is  necessary.  First,  the  results  obtained  in  Chapter  IV  are 
based  upon  an  arbitrary  grouping  of  CRD  As.  Second,  the  CRD  As  were  from  Wright 
Laboratory  (WL)  which  has  since  been  incorporated  into  the  Air  Force  Research 
Laboratory  (AFRL).  As  such,  the  directorates  examined  in  Chapter  IV  are  now  dispersed 
into  different  parts  of  AFRL.  Nevertheless,  the  following  are  several  suggestions  for 
follow-on  research: 

1 .  Expand  the  Model;  The  model  illustrated  in  this  research  is  concerned  solely  with 
the  benefits  experienced  by  the  government  in  technology  transfer.  It  has  been  shown 
in  Chapter  I  that  there  are  nongovernmental  benefits  such  as  the  stimulation  of  the 
economy,  etc.  These  benefits  were  examined  in  the  AFMC  TTO  1994  paper  entitled 
“Technology  Transfer:  The  PayoffBenefits’  Indicator.”  Using  these  two  research 
sources  as  a  starting  point,  both  governmental  and  nongovernmental  benefits  could  be 
examined  and  the  model  broadened. 

2.  Sensitivity  analysis;  This  research  effort  performed  sensitivity  analysis  by  varying  the 
relative  importance  of  objective  benefits  to  subjective  benefits  (a).  Further  sensitivity 
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analysis  could  be  performed  by  varying  the  unitized  weights  (relative  importance  of 
subjective  benefits  to  one  another). 

3.  Other  Possibilities:  Other  areas  of  possible  follow-on  research  include  applying  the 
methodology  to  AFRL  CRD  As.  The  collection  of  non-proprietary  data  would  enable 
validation  of  results,  which  could  be  accomplished  through  the  comparison  of  expert 
opinions  to  model  results.  A  final  option  is  to  apply  the  procedure  to  another 
government  organization  such  as  the  Department  of  Energy. 

Summary 

Air  Force  technology  transfer  began  in  the  early  1980’s  when  it  was  realized  that 
technology  transfer  could  provide  benefits  such  as  lower  R&D  costs  to  government  and 
commercial  industry  participants  alike.  Over  the  past  two  decades,  many  advances  to  the 
technology  transfer  process  have  been  made  (e.g.,  development  of  CRD  As).  This 
research  effort  improves  the  process  by  introducing  a  model  to  determine  return  on 
investment.  The  model  considers  both  the  objective  and  subjective  benefits  and  helps 
decision-makers  choose  between  potential  alternatives.  This  choice  is  not  limited  to 
comparing  benefits  as  the  model’s  sensitivity  analysis  capability  enables  choices  to  be 
made  based  on  the  decision-maker’s  perspective  concerning  each  benefit’s  importance. 
The  model  not  only  fills  a  current  void  in  technology  transfer,  it  champions  transferring 
those  technologies  that  are  most  beneficial.  The  model,  together  with  the  opportunities 
for  improvement  listed  in  the  future  research  recommendation’s  section,  will  improve  the 
technology  transfer  process.  Technology  transfer  and  its  continued  improvement  are  vital 
as  the  United  States  attempts  to  maintain  a  technological  edge. 
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Appendix  A:  CRDA  Benefits  Questionnaire 


Background  Information  (Print  responses): 

1.  Name/Office  Symbol:  _ _ _ _ _ _ _ 

2.  Position  Title:  _ _ _ _ _ _ 

3.  CRDA  Number: _ _ _ — — - 

4.  CRDA  Title:  _ _ _ _ 

Quantitative  Benefits  (Place  x  in  box  for  yes  or  no,  Print  dollar  amounts  as  appropriate, 

Include  estimates  of  future  benefits  that  are  likely  to  occur  within  2  years  or  less): 

5.  a.  Did  the  government  receive  any  kind  of  royalty  payments  as  a  direct  result  of  this  CRDA? 

□  Yes  □  No 

b.  If  yes,  estimate  the  total  amount  of  royalty  payments? 

To  date:  $ _  Future:  $ _ 

6.  a.  Did  the  government  receive  any  kind  of  cash  reimbursement  for  services  performed  by  the 
laboratory  as  a  direct  result  of  this  CRDA  (e.g.,  facilities  payments  or  salary  payments)? 

□  Yes  □  No 

b.  If  yes,  estimate  the  total  amount  of  all  cash  reimbursements? 

To  date:  $ _  Future:  $ _ 

7.  a.  Did  the  government  receive  any  kind  of  license  fees  as  a  direct  result  of  this  CRDA? 

□  Yes  □  No 

b.  If  yes,  estimate  the  total  amount  of  license  fees? 

To  date:  $_ _  Future:  $ _ 

8.  a.  Did  the  government  realize  any  financial  benefits  that  are  not  directly  payable  as  cash  as  a  direct 
result  of  this  CRDA? 

□  Yes  □  No 

b.  If  yes,  estimate  the  total  amount  of  cost  savings  for  each  category? 


Work  Avoidance: 

To  date:  $ 

Future: 

Data  Received: 

To  date:  S 

Future: 

Software  Received: 

To  date:  $ 

Future: 

Hardware  Received: 

To  date:  $ 

Future: 

Other 

To  date:  S 

Future: 
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Qualitative  Benefits  (Place  x  in  box  for  yes  or  no.  If  yes  then  Circle  the  most  appropriate  response 
for  each  part  b): 

9.  a.  Did  this  CRDA  result  in  productivity  increases  in  the  laboratory  (e.g.,  increased  automation)? 

□  Yes  □  No 

b  If  yes  then  rate  the  importance  of  these  productivity  increases  to  the  laboratory. 

1  2  3  4  *  , 

not  slightly  moderately  very  extremely 

important  important  important  important  important 

10.  a.  Did  this  CRDA  result  in  better  utilization  of  laboratory  resources  (e.g.,  more  use  of  a 

laboratory  test  facility)? 

□  Yes  □  No 

b.  If  yes,  then  rate  the  importance  of  this  better  utilization  of  resources  to  the  laboratory. 

1  2  3  4  5 

not  slightly  moderately  very  extremely 

important  important  important  important  important 


11.  a.  Did  this  CRDA  result  in  improved  management  and  business  practices  in  the  laboratory  (e.g., 
improved  skills  and  techniques  through  interaction  with  the  partner)? 

□  Yes  □  No  . 

b.  If  yes,  then  rate  the  importance  of  these  improved  management  and  business  practices  to  the 

laboratory. 

1  2  3  4  5 

not  slightly  moderately  very  extremely 

important  important  important  important  important 

12.  a.  Did  this  CRDA  result  in  improved  laboratory  image? 

□  Yes  □  No 

b  Ifves  then  rate  the  importance  of  this  improved  laboratory  image  to  the  laboratory. 

’j  2  3  4  5 

not  slightly  moderately  very  extremely 

important  important  important  important  important 


13. 


14. 


a.  Did  this  CRDA  result  in  improved  morale  in  the  laboratory? 

□  Yes  □  No 

b.  If  yes,  then  rate  the  importance  of  this  improved  morale  to  the  laboratory. 

1  2  3  4  5 

not  slightly  moderately  very  extremely 

important  important  important  important  important 


a.  Did  this  CRDA  result  in  improved  technical  capability  in  the  laboratory? 

□  Yes  □  No 

b.  If  yes,  then  rate  the  importance  of  this  improved  technical  capability  to  the  laboratory. 

1  2  3  4  5 

not  slightly  moderately  very  extremely 


important  important  important 


important  important 
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Appendix  B:  Technology  Transfer  Return  on  Investment  Questionnaire 

Previous  research  has  indicated  that  technology  transfer  benefits  are  both  objective  and  subjective  in 
nature.  The  purpose  of  my  research  is  to  determine  a  model  that  enables  one  to  determine  which  transfer 
opportunities  (CRD As)  are  most  desirable.  In  order  to  develop  a  ROI  model  concerning  technology 
transfer,  I  need  to  understand  which  benefits  are  most  important  to  decision-makers.  The  three  sections 
below  contain  possible  benefits  associated  with  a  hypothetical  CRDA  (subjective  and  objective  benefits). 
Each  section  contains  the  definitions  for  each  benefit  and  then  provides  an  example  of  how  to  provide 
responses  indicating  the  importance  you  place  on  each  benefit  in  comparison  to  each  other.  Your  task  is 
to  provide  your  rating  of  each  benefit  in  comparison  to  each  other. 

SECTION  1  -  SUBJECTIVE  BENEFITS: 

1.  Productivity  -  Did  this  CRDA  result  in  productivity  increases  in  the  laboratory  (e.g.,  is  the  lab  more 
efficient  in  its  technology  development)? 

2.  Resource  utilization  -  Did  this  CRDA  result  in  better  utilization  of  specialized  laboratory  resources 
(e.g.,  more  use  of  a  laboratory  test  facility)? 

3.  Management  -  Did  this  CRDA  result  in  improved  management  and  business  practices  in  the 
laboratory  (e.g.,  improved  skills  and  techniques  through  interaction  with  the  partner)? 

4.  image  -  Did  this  CRDA  result  in  improved  laboratory  image? 

5.  Morale-  Did  this  CRDA  result  in  improved  morale  in  the  laboratory? 

6.  Technical  Capability-  Did  this  CRDA  improve  lab  personnel  technical  capability? 

Given  the  above  definitions,  please  rate  the  individual  subjective  benefits  with  respect  to  each  other. 
Specifically,  how  much  (of  a  possible  100  points)  is  each  subjective  benefit  of  value  to  you  as  a  manager 
(NOTE:  the  6  values  must  sum  to  100).  An  example  response  is  provided. 


EXAMPLE  RESPONSE 

YOUR  RESPONSE 

Productivity 

12 

Productivity 

_ 

Resource  utilization 

25 

Resource  utilization 

_ 

Management 

15 

Management 

— 

Image 

_5 

Image 

— 

Morale 

20 

Morale 

— 

Technical  Caoabilitv 

23 

Technical  CaDabilitv 

_ 

TOTAL 

100 

TOTAL 

100 
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SECTION  2  -  OBJECTIVE  BENEFITS: 

1.  Royalties  -  Did  the  government  receive  any  kind  of  royalty  payment  as  a  direct  result  of  this  CRD  A? 

2  rash  reimbursement  -  Did  the  government  receive  any  kind  of  cash  reimbursement  for 
services  performed  by  the  laboratory  as  a  direct  result  of  this  CRDA? 

2.  t  tr-p.nse  fees  -  Did  the  government  receive  any  kind  of  license  fees  as  a  direct  result  of  this 
CRDA? 

1  Wnrv  avniHanra  -  Did  the  government  realize  any  financial  benefits  that  are  not  directly  payable  as 
cash  as  a  direct  result  of  this  CRDA:  specifically  work  avoidance? 

2.  Data  received  -  Did  the  government  realize  any  financial  benefits  that  are  not  directly  payable  as  cash 
as  a  direct  result  of  this  CRDA:  specifically  data  received? 

3.  Software  received  -  Did  the  government  realize  any  financial  benefits  that  are  not  directly  payable  as 
cash  as  a  direct  result  of  this  CRDA:  specifically  software  received? 

4  Hardware  received  -  Did  the  government  realize  any  financial  benefits  that  are  not  directly  payable  as 
cash  as  a  direct  result  of  this  CRDA:  specifically  hardware  received? 

Given  the  above  definitions,  please  rate  the  individual  objective  benefits  with  respect  to  each  other. 

Specifically,  how  much  (of  a  possible  100  points)  is  each  objective  benefit  of  value  to  you  as  a  manager 

(NOTE:  the  7  values  must  sum  to  100).  An  example  response  is  provided. 


F.Y AMPLE  RESPONSE 

YOTTR  RESPONSE 

Royalty  payments 

25 

Royalty  payments 

— 

Cash  reimbursements 

25 

Cash  reimbursements 

— 

License  fees 

15 

License  fees 

— 

Work  avoidance 

15 

Work  avoidance 

— 

Data  received 

_5 

Data  received 

— 

Software  received 

_5 

Software  received 

— 

Hardware  received 

10 

Hardware  received 

— 

TOTAL 

100 

TOTAL 

100 

SECTION  3  -  OBJECTIVE  VERSUS  SUBJECTIVE  BENEFITS: 

As  a  manager,  what  is  your  overall  impression  of  the  importance  of  objective  versus  subjective 
benefits?  The  example  below  shows  objective  benefits  being  3  times  more  important. 

VOTTR  RESPONSE 

Objective  benefits  _ 

Subjective  benefits  _ 

TOTAL  100 


Objective  benefits 
Subjective  benefits 

TOTAL 


75 

25 

100 
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Appendix  C:  CKDA  Worksheets 
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